System and Method for Controlling Vehicle Headlights

ABSTRACT

Control system and method for automatically controlling headlights of a vehicle includes an optical system for imaging external sources of light within a predetermined field of view and an image processing system for processing images from the optical system and providing a control signal for controlling the headlights as a function of the processed images. Processing the images may entail identifying a source of radiation in the images, the control signal being provided to dim the headlights when a source of radiation in the images is identified as a headlight or taillight of another vehicle. To this end, the image processing system may include a trained pattern recognition system for processing the images to identify the source of radiation, e.g., a pattern recognition algorithm generated from data of possible sources of radiation including headlights and taillights of vehicles and patterns of received radiation from the possible sources.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is:

1. a continuation-in-part (CIP) of U.S. patent application Ser. No.10/413,426 filed Apr. 14, 2003, which is:

A. a CIP of U.S. patent application Ser. No. 09/765,559 filed Jan. 19,2001, now U.S. Pat. No. 6,553,296, which is a CIP of U.S. patentapplication Ser. No. 09/476,255 filed Dec. 30, 1999, now U.S. Pat. No.6,324,453, which claims priority under 35 U.S.C. §119(e) of U.S.provisional patent application Ser. No. 60/114,507 filed Dec. 31, 1998;and

B. a CIP of U.S. patent application Ser. No. 09/925,043 filed Aug. 8,2001, now U.S. Pat. No. 6,507,779, which is:

-   -   1. a CIP of U.S. patent application Ser. No. 09/765,559 filed        Jan. 19, 2001, now U.S. Pat. No. 6,553,296, the history of which        is set forth above; and    -   2. a CIP of U.S. patent application Ser. No. 09/389,947 filed        Sep. 3, 1999, now U.S. Pat. No. 6,393,133, which is a CIP of        U.S. patent application Ser. No. 09/200,614, filed Nov. 30,        1998, now U.S. Pat. No. 6,141,432, which is a continuation of        U.S. patent application Ser. No. 08/474,786 filed Jun. 7, 1995,        now U.S. Pat. No. 5,845,000;

C. a CIP of U.S. patent application Ser. No. 10/116,808 filed Apr. 5,2002, now U.S. Pat. No. 6,856,873, which is a CIP of U.S. patentapplication Ser. No. 09/838,919 filed Apr. 20, 2001, now U.S. Pat. No.6,442,465, which is a CIP of U.S. patent application Ser. No. 09/389,947filed Sep. 3, 1999, now U.S. Pat. No. 6,393,133, the history of which isset forth above; and

D. a CIP of U.S. patent application Ser. No. 10/302,105 filed Nov. 22,2002, now U.S. Pat. No. 6,772,057;

2. a CIP of U.S. patent application Ser. No. 10/895,121 filed Jul. 21,2004 which is a continuation of U.S. patent application Ser. No.10/733,957 filed Dec. 11, 2003, now U.S. Pat. No. 7,243,945, which is:

A. a CIP of U.S. patent application Ser. No. 10/116,808 filed Apr. 5,2002, now U.S. Pat. No. 6,856,873, the history of which is set forthabove;

B. a CIP of U.S. patent application Ser. No. 10/302,105 filed Nov. 22,2002, now U.S. Pat. No. 6,772,057, the history of which is set forthabove;

3. a CIP of U.S. patent application Ser. No. 10/940,881 filed Sep. 13,2004 which is a CIP of U.S. patent application Ser. No. 10/116,808 filedApr. 5, 2002, now U.S. Pat. No. 6,856,873, the history of which is setforth above;4. a CIP of U.S. patent application Ser. No. 11/025,501 filed Jan. 3,2005 which is a CIP of U.S. patent application Ser. No. 10/116,808 filedApr. 5, 2002, now U.S. Pat. No. 6,856,873, the history of which is setforth above;5. a CIP of U.S. patent application Ser. No. 11/455,497 filed Jun. 19,2006;6. a CIP of U.S. patent application Ser. No. 11/502,039 filed Aug. 10,2006;7. a CIP of U.S. patent application Ser. No. 11/538,934 filed Oct. 5,2006; and8. a CIP of U.S. patent application Ser. No. 11/558,996 filed Nov. 13,2006.

This application is related to U.S. patent application Ser. No.08/474,782 filed Jun. 7, 1995, now U.S. Pat. No. 5,835,613, and10/931,288 filed Aug. 31, 2004, now U.S. Pat. No. 7,164,117, on thegrounds that they include common subject matter.

All of the above-mentioned applications are incorporated by referenceherein.

FIELD OF THE INVENTION

The present invention relates generally to systems and methods forcontrolling headlights of a vehicle, and more specifically, toselectively and temporarily dimming the headlights when other vehiclesin the field of the view of the headlights are detected.

BACKGROUND OF THE INVENTION

Background of the invention is found in the parent applications, inparticular the '786 application. All of the patents, patentapplications, technical papers and other references mentioned below andin the parent applications are incorporated herein by reference in theirentirety.

Possible definitions of terms used in the application are set forth inthe '881 application, incorporated by reference herein.

Preferred embodiments of the invention are described below and unlessspecifically noted, it is the applicant's intention that the words andphrases in the specification and claims be given the ordinary andaccustomed meaning to those of ordinary skill in the applicable art(s).If the applicant intends any other meaning, he will specifically statehe is applying a special meaning to a word or phrase.

Likewise, applicant's use of the word “function” here is not intended toindicate that the applicant seeks to invoke the special provisions of 35U.S.C. §112, sixth paragraph, to define his invention. To the contrary,if applicant wishes to invoke the provisions of 35 U.S.C. §112, sixthparagraph, to define his invention, he will specifically set forth inthe claims the phrases “means for” or “step for” and a function, withoutalso reciting in that phrase any structure, material or act in supportof the function. Moreover, even if applicant invokes the provisions of35 U.S.C. §112, sixth paragraph, to define his invention, it is theapplicant's intention that his inventions not be limited to the specificstructure, material or acts that are described in the preferredembodiments herein. Rather, if applicant claims his inventions byspecifically invoking the provisions of 35 U.S.C. §112, sixth paragraph,it is nonetheless his intention to cover and include any and allstructure, materials or acts that perform the claimed function, alongwith any and all known or later developed equivalent structures,materials or acts for performing the claimed function.

OBJECTS AND SUMMARY OF THE INVENTION

It is an object of the present invention to provide new and improvedsystems and methods for controlling headlights of a vehicle, and morespecifically, to selectively and temporarily dimming the headlights whenother vehicles in the field of the view of the headlights are detected.

It is another object of the present invention to control vehicularsystems, including a headlight dimming system, based on analysis ofimages obtained from one or more rear-view mirror mounted imagingdevices.

In order to achieve one or both of these objects, and possibly others, acontrol system for automatically controlling headlights of a vehicle inaccordance with the invention includes an optical system for imagingexternal sources of light within a predetermined field of view and animage processing system for processing images from the optical systemand providing a control signal for controlling the headlights as afunction of the processed images. Processing the images may entailidentifying a source of radiation in the images, the control signalbeing provided to dim the headlights when a source of radiation in theimages is identified as a headlight or taillight of another vehicle. Tothis end, the image processing system may include a trained patternrecognition system for processing the images to identify the source ofradiation, e.g., a pattern recognition algorithm generated from data ofpossible sources of radiation including headlights and taillights ofvehicles and patterns of received radiation from the possible sources.The trained pattern recognition algorithm may comprise a neural network.

The optical system may comprise a CCD array. It may be arranged on arear view mirror in an interior of the vehicle. It may be arranged on apart of the vehicle that is not movable relative to a frame of thevehicle, i.e., a fixed part of the inside rear view mirror. The opticalsystem may include an image array sensor containing a plurality ofpixels. It may include a plurality of image array sensors.

A method for automatically controlling headlights of a vehicle inaccordance with the invention includes imaging external sources of lightwithin a predetermined field of view to obtain images of an environmentoutside of the vehicle, processing the obtained images from the opticalsystem, and providing a control signal for controlling the headlights asa function of the processed images. The same features of the systemdescribed above may be applied to the method. For example, processingthe images may entail training a pattern recognition algorithm toidentify the source of radiation in a training stage in which knownsources of radiation are provided, images including these known sourcesare obtained and the pattern recognition is formed based on theassociation of the known sources with the obtained images.

The invention also includes a vehicle including headlights, a rear viewmirror situated inside a passenger compartment of the vehicle, and acontrol system for automatically controlling the headlights of avehicle. The control system includes an optical system for imagingexternal sources of light and which is arranged on the rear view mirror,and an image processing system for receiving images from the opticalsystem and identifying whether sources of light in the images originatefrom headlights or taillights of other vehicles. The image processingsystem is arranged to dim the headlights when it identifies that asource of light in an image originates from the headlights or taillightsof another vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

The following drawings are illustrative of embodiments of the systemdeveloped or adapted using the teachings of at least one of theinventions disclosed herein and are not meant to limit the scope of theinvention as encompassed by the claims. In particular, the illustrationsbelow are frequently limited to the monitoring of the front passengerseat for the purpose of describing the system. The invention applies aswell to adapting the system to the other seating positions in thevehicle and particularly to the driver and rear passenger positions.

FIG. 1 is a side view with parts cutaway and removed of a vehicleshowing the passenger compartment containing a rear facing child seat onthe front passenger seat and a preferred mounting location for anoccupant and rear facing child seat presence detector including anantenna field sensor and a resonator or reflector placed onto theforward most portion of the child seat.

FIG. 2 is a perspective view of a vehicle showing the position of theultrasonic or electromagnetic sensors relative to the driver and frontpassenger seats.

FIG. 3 is a side view with parts cutaway and removed of a vehicleshowing the passenger compartment containing a box on the frontpassenger seat and a preferred mounting location for an occupant andrear facing child seat presence detector and including an antenna fieldsensor.

FIG. 4 is a side view with parts cutaway and removed of a vehicleshowing the passenger compartment containing a driver and a preferredmounting location for an occupant identification system and including anantenna field sensor and an inattentiveness response button.

FIG. 5 is a side view, with certain portions removed or cut away, of aportion of the passenger compartment of a vehicle showing severalpreferred mounting locations of occupant position sensors for sensingthe position of the vehicle driver.

FIG. 6 shows a seated-state detecting unit in accordance with thepresent invention and the connections between ultrasonic orelectromagnetic sensors, a weight sensor, a reclining angle detectingsensor, a seat track position detecting sensor, a heartbeat sensor, amotion sensor, a neural network, and an airbag system installed within avehicular compartment.

FIG. 6A is an illustration as in FIG. 6 with the replacement of a straingage weight sensor within a cavity within the seat cushion for thebladder weight sensor of FIG. 6.

FIG. 7 is a flow chart of the environment monitoring in accordance withthe invention.

FIG. 8A is a side planar view, with certain portions removed or cutaway, of a portion of the passenger compartment of a vehicle showingseveral preferred mounting locations of interior vehicle monitoringsensors shown particularly for sensing the vehicle driver illustratingthe wave pattern from a CCD or CMOS optical position sensor mountedalong the side of the driver or centered above his or her head.

FIG. 8B is a view as in FIG. 8A illustrating the wave pattern from anoptical system using an infrared light source and a CCD or CMOS arrayreceiver using the windshield as a reflection surface and showingschematically the interface between the vehicle interior monitoringsystem of at least one of the inventions disclosed herein and aninstrument panel mounted inattentiveness warning light or buzzer andreset button.

FIG. 8C is a view as in FIG. 8A illustrating the wave pattern from anoptical system using an infrared light source and a CCD or CMOS arrayreceiver where the CCD or CMOS array receiver is covered by a lenspermitting a wide angle view of the contents of the passengercompartment.

FIG. 8D is a view as in FIG. 8A illustrating the wave pattern from apair of small CCD or CMOS array receivers and one infrared transmitterwhere the spacing of the CCD or CMOS arrays permits an accuratemeasurement of the distance to features on the occupant.

FIG. 8E is a view as in FIG. 8A illustrating the wave pattern from a setof ultrasonic transmitter/receivers where the spacing of the transducersand the phase of the signal permits an accurate focusing of theultrasonic beam and thus the accurate measurement of a particular pointon the surface of the driver.

FIGS. 9A and 9B are functional block diagrams of the ultrasonic imagingsystem illustrated in FIG. 1 using a microprocessor, DSP or fieldprogrammable gate array (FGPA) (FIG. 9A) or an application specificintegrated circuit (ASIC) (FIG. 9B).

FIG. 10 is a perspective view of a vehicle about to impact the side ofanother vehicle showing the location of the various parts of theanticipatory sensor system of at least one of the inventions disclosedherein.

FIG. 11 is a schematic illustration of the exterior monitoring system inaccordance with the invention.

FIG. 12 is a side planar view, with certain portions removed or cutaway, of a portion of the passenger compartment illustrating a sensorfor sensing the headlights of an oncoming vehicle and/or the taillightsof a leading vehicle used in conjunction with an automatic headlightdimming system.

FIG. 13 is a schematic illustration of the position measuring inaccordance with the invention.

FIG. 14 is a schematic illustrating the circuit of an occupantposition-sensing device using a modulated infrared signal, beatfrequency and phase detector system.

FIG. 15 a flowchart showing the training steps of a neural network.

FIG. 16 is a schematic illustration of a system for controllingoperation of a vehicle or a component thereof based on recognition of anauthorized individual.

FIG. 17 is a schematic illustration of the environment monitoring inaccordance with the invention.

FIG. 18 is a diagram showing an example of an occupant sensing strategyfor a single camera optical system.

FIG. 19 is a processing block diagram of the example of FIG. 18.

FIG. 20 is a block diagram of an antenna-based near field objectdiscriminator.

FIG. 21 is a side view, with certain portions removed or cut away, of aportion of the passenger compartment of a vehicle showing preferredmounting locations of optical interior vehicle monitoring sensors

FIG. 22 is a side view with parts cutaway and removed of a subjectvehicle and an oncoming vehicle, showing the headlights of the oncomingvehicle and the passenger compartment of the subject vehicle, containingdetectors of the driver's eyes and detectors for the headlights of theoncoming vehicle and the selective filtering of the light of theapproaching vehicle's headlights through the use of electro-chromicglass, organic or metallic semiconductor polymers or electrophericparticulates (SPD) in the windshield.

FIG. 22A is an enlarged view of the section 22A in FIG. 22.

FIG. 23 is a side view with parts cutaway and removed of a vehicle and afollowing vehicle showing the headlights of the following vehicle andthe passenger compartment of the leading vehicle containing a driver anda preferred mounting location for driver eyes and following vehicleheadlight detectors and the selective filtering of the light of thefollowing vehicle's headlights through the use of electrochromic glass,SPD glass or equivalent, in the rear view mirror.

FIG. 23A is an enlarged view of the section designated 23A in FIG. 23.

FIG. 23B is an enlarged view of the section designated 23B in FIG. 23A.

FIG. 24 illustrates the interior of a passenger compartment with a rearview mirror, a camera for viewing the eyes of the driver and a largegenerally transparent visor for glare filtering.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

A patent or literature referred to below is incorporated by reference inits entirety. Also, although many of the examples below relate to aparticular vehicle, an automobile, the invention is not limited to anyparticular vehicle and is thus applicable to all relevant vehiclesincluding shipping containers and truck trailers and to all compartmentsof a vehicle including, for example, the passenger compartment and thetrunk of an automobile or truck.

1. General Occupant Sensors

Referring to the accompanying drawings, FIG. 1 is a side view, withparts cutaway and removed of a vehicle showing the passengercompartment, or passenger container, containing a rear facing child seat2 on a front passenger seat 4 and a preferred mounting location for afirst embodiment of a vehicle interior monitoring system in accordancewith the invention. The interior monitoring system is capable ofdetecting the presence of an object, occupying objects such as a box, anoccupant or a rear facing child seat 2, determining the type of object,determining the location of the object, and/or determining anotherproperty or characteristic of the object. A property of the object couldbe the orientation of a child seat, the velocity of an adult and thelike. For example, the vehicle interior monitoring system can determinethat an object is present on the seat, that the object is a child seatand that the child seat is rear-facing. The vehicle interior monitoringsystem could also determine that the object is an adult, that he isdrunk and that he is out of position relative to the airbag.

In this embodiment, three transducers 6, 8 and 10 are used alone, or,alternately in combination with one or more antenna near fieldmonitoring sensors or transducers, 12, 14 and 16, although any number ofwave-transmitting transducers or radiation-receiving receivers may beused. Such transducers or receivers may be of the type that emit orreceive a continuous signal, a time varying signal or a spatial varyingsignal such as in a scanning system and each may comprise only atransmitter which transmits energy, waves or radiation, only a receiverwhich receives energy, waves or radiation, both a transmitter and areceiver capable of transmitting and receiving energy, waves orradiation, an electric field sensor, a capacitive sensor, or aself-tuning antenna-based sensor, weight sensor, chemical sensor, motionsensor or vibration sensor, for example.

One particular type of radiation-receiving receiver for use in theinvention receives electromagnetic waves and another receives ultrasonicwaves.

In an ultrasonic embodiment, transducer 8 can be used as a transmitterand transducers 6 and 10 can be used as receivers. Other combinationscan be used such as where all transducers are transceivers (transmittersand receivers). For example, transducer 8 can be constructed to transmitultrasonic energy toward the front passenger seat, which is modified, inthis case by the occupying item of the passenger seat, i.e., the rearfacing child seat 2, and the modified waves are received by thetransducers 6 and 10, for example. A more common arrangement is wheretransducers 6, 8 and 10 are all transceivers. Modification of theultrasonic energy may constitute reflection of the ultrasonic energy asthe ultrasonic energy is reflected back by the occupying item of theseat. The waves received by transducers 6 and 10 vary with timedepending on the shape of the object occupying the passenger seat, inthis case the rear facing child seat 2. Each different occupying itemwill reflect back waves having a different pattern. Also, the pattern ofwaves received by transducer 6 will differ from the pattern received bytransducer 10 in view of its different mounting location. Thisdifference generally permits the determination of location of thereflecting surface (i.e., the rear facing child seat 2) throughtriangulation. Through the use of two transducers 6, 10, a sort ofstereographic image is received by the two transducers and recorded foranalysis by processor 20, which is coupled to the transducers 6, 8, 10,e.g., by wires or wirelessly. This image will differ for each objectthat is placed on the vehicle seat and it will also change for eachposition of a particular object and for each position of the vehicleseat. Elements 6, 8, 10, although described as transducers, arerepresentative of any type of component used in a wave-based analysistechnique. Also, although the example of an automobile passengercompartment has been shown, the same principle can be used formonitoring the interior of any vehicle including in particular shippingcontainers and truck trailers.

Wave-type sensors as the transducers 6, 8, 10 as well as electric fieldsensors 12, 14, 16 are mentioned above. Electric field sensors and wavesensors are essentially the same from the point of view of sensing thepresence of an occupant in a vehicle. In both cases, a time varyingelectric field is disturbed or modified by the presence of the occupant.At high frequencies in the visual, infrared and high frequency radiowave region, the sensor is based on its capability to sense a change ofwave characteristics of the electromagnetic field, such as amplitude,phase or frequency. As the frequency drops, other characteristics of thefield are measured. At still lower frequencies, the occupant'sdielectric properties modify parameters of the reactive electric fieldin the occupied space between or near the plates of a capacitor. In thislatter case, the sensor senses the change in charge distribution on thecapacitor plates by measuring, for example, the current wave magnitudeor phase in the electric circuit that drives the capacitor. Thesemeasured parameters are directly connected with parameters of thedisplacement current in the occupied space. In all cases, the presenceof the occupant reflects, absorbs or modifies the waves or variations inthe electric field in the space occupied by the occupant. Thus, for thepurposes of at least one of the inventions disclosed herein,capacitance, electric field or electromagnetic wave sensors areequivalent and although they are all technically “field” sensors theywill be considered as “wave” sensors herein. What follows is adiscussion comparing the similarities and differences between two typesof field or wave sensors, electromagnetic wave sensors and capacitivesensors as exemplified by Kithil in U.S. Pat. No. 5,702,634.

An electromagnetic field disturbed or emitted by a passenger in the caseof an electromagnetic wave sensor, for example, and the electric fieldsensor of Kithil, for example, are in many ways similar and equivalentfor the purposes of at least one of the inventions disclosed herein. Theelectromagnetic wave sensor is an actual electromagnetic wave sensor bydefinition because they sense parameters of an electromagnetic wave,which is a coupled pair of continuously changing electric and magneticfields. The electric field here is not a static, potential one. It isessentially a dynamic, rotational electric field coupled with a changingmagnetic one, that is, an electromagnetic wave. It cannot be produced bya steady distribution of electric charges. It is initially produced bymoving electric charges in a transmitter, even if this transmitter is apassenger body for the case of a passive infrared sensor.

In the Kithil sensor, a static electric field is declared as an initialmaterial agent coupling a passenger and a sensor (see Column 5, lines5-7: “The proximity sensor 12 each function by creating an electrostaticfield between oscillator input loop 54 and detector output loop 56,which is affected by presence of a person near by, as a result ofcapacitive coupling, . . . ”). It is a potential, non-rotationalelectric field. It is not necessarily coupled with any magnetic field.It is the electric field of a capacitor. It can be produced with asteady distribution of electric charges. Thus, it is not anelectromagnetic wave by definition but if the sensor is driven by avarying current, then it produces a quasistatic electric field in thespace between/near the plates of the capacitor.

Kithil declares that his capacitance sensor uses a static electricfield. Thus, from the consideration above, one can conclude thatKithil's sensor cannot be treated as a wave sensor because there are noactual electromagnetic waves but only a static electric field of thecapacitor in the sensor system. However, this is not believed to be thecase. The Kithil system could not operate with a true static electricfield because a steady system does not carry any information. Therefore,Kithil is forced to use an oscillator, causing an alternate current inthe capacitor and a reactive quasi-static electric field in the spacebetween the capacitor plates, and a detector to reveal an informativechange of the sensor capacitance caused by the presence of an occupant(see FIG. 2 and its description). In this case, the system becomes a“wave sensor” in the sense that it starts generating an actualtime-varying electric field that certainly originates electromagneticwaves according to the definition above. That is, Kithil's sensor can betreated as a wave sensor regardless of the shape of the electric fieldthat it creates, a beam or a spread shape.

As follows from the Kithil patent, the capacitor sensor is likely aparametric system where the capacitance of the sensor is controlled bythe influence of the passenger body. This influence is transferred bymeans of the near electromagnetic field (i.e., the wave-like process)coupling the capacitor electrodes and the body. It is important to notethat the same influence takes place with a real static electric fieldalso, that is in absence of any wave phenomenon. This would be asituation if there were no oscillator in Kithil's system. However, sucha system is not workable and thus Kithil reverts to a dynamic systemusing time-varying electric fields.

Thus, although Kithil declares that the coupling is due to a staticelectric field, such a situation is not realized in his system becausean alternating electromagnetic field (“quasi-wave”) exists in the systemdue to the oscillator. Thus, his sensor is actually a wave sensor, thatis, it is sensitive to a change of a wave field in the vehicularcompartment. This change is measured by measuring the change of itscapacitance. The capacitance of the sensor system is determined by theconfiguration of its electrodes, one of which is a human body, that is,the passenger inside of and the part which controls the electrodeconfiguration and hence a sensor parameter, the capacitance.

The physics definition of “wave” from Webster's Encyclopedic UnabridgedDictionary is: “11. Physics. A progressive disturbance propagated frompoint to point in a medium or space without progress or advance of thepoints themselves, . . . ”. In a capacitor, the time that it takes forthe disturbance (a change in voltage) to propagate through space, thedielectric and to the opposite plate is generally small and neglectedbut it is not zero. As the frequency driving the capacitor increases andthe distance separating the plates increases, this transmission time asa percentage of the period of oscillation can become significant.Nevertheless, an observer between the plates will see the rise and fallof the electric field much like a person standing in the water of anocean. The presence of a dielectric body between the plates causes thewaves to get bigger as more electrons flow to and from the plates of thecapacitor. Thus, an occupant affects the magnitude of these waves whichis sensed by the capacitor circuit. Thus, the electromagnetic field is amaterial agent that carries information about a passenger's position inboth Kithil's and a beam-type electromagnetic wave sensor.

For ultrasonic systems, the “image” recorded from each ultrasonictransducer/receiver, is actually a time series of digitized data of theamplitude of the received signal versus time. Since there are tworeceivers, two time series are obtained which are processed by theprocessor 20. The processor 20 may include electronic circuitry andassociated, embedded software. Processor 20 constitutes one form ofgenerating means in accordance with the invention which generatesinformation about the occupancy of the passenger compartment based onthe waves received by the transducers 6, 8, 10.

When different objects are placed on the front passenger seat, theimages from transducers 6, 8, 10 for example, are different but thereare also similarities between all images of rear facing child seats, forexample, regardless of where on the vehicle seat it is placed andregardless of what company manufactured the child seat. Alternately,there will be similarities between all images of people sitting on theseat regardless of what they are wearing, their age or size. The problemis to find the “rules” which differentiate the images of one type ofobject from the images of other types of objects, e.g., whichdifferentiate the occupant images from the rear facing child seatimages. The similarities of these images for various child seats arefrequently not obvious to a person looking at plots of the time seriesand thus computer algorithms are developed to sort out the variouspatterns. For a more detailed discussion of pattern recognition see U.S.Pat. No. RE 37,260.

The determination of these rules is important to the pattern recognitiontechniques used in at least one of the inventions disclosed herein. Ingeneral, three approaches have been useful, artificial intelligence,fuzzy logic and artificial neural networks (including cellular andmodular or combination neural networks and support vectormachines—although additional types of pattern recognition techniques mayalso be used, such as sensor fusion). In some implementations of atleast one of the inventions disclosed herein, such as the determinationthat there is an object in the path of a closing window as describedbelow, the rules are sufficiently obvious that a trained researcher cansometimes look at the returned signals and devise a simple algorithm tomake the required determinations. In others, such as the determinationof the presence of a rear facing child seat or of an occupant,artificial neural networks can be used to determine the rules. One suchset of neural network software for determining the pattern recognitionrules is available from the International Scientific Research, Inc. ofPanama City, Panama.

Electromagnetic energy based occupant sensors exist that use manyportions of the electromagnetic spectrum. A system based on theultraviolet, visible or infrared portions of the spectrum generallyoperate with a transmitter and a receiver of reflected radiation. Thereceiver may be a camera or a photo detector such as a pin or avalanchediode as described in above-referenced patents and patent applications.At other frequencies, the absorption of the electromagnetic energy isprimarily used and at still other frequencies the capacitance orelectric field influencing effects are used. Generally, the human bodywill reflect, scatter, absorb or transmit electromagnetic energy invarious degrees depending on the frequency of the electromagnetic waves.All such occupant sensors are included herein.

In an embodiment wherein electromagnetic energy is used, it is to beappreciated that any portion of the electromagnetic signals thatimpinges upon, surrounds or involves a body portion of the occupant isat least partially absorbed by the body portion. Sometimes, this is dueto the fact that the human body is composed primarily of water, and thatelectromagnetic energy of certain frequencies is readily absorbed bywater. The amount of electromagnetic signal absorption is related to thefrequency of the signal, and size or bulk of the body portion that thesignal impinges upon. For example, a torso of a human body tends toabsorb a greater percentage of electromagnetic energy than a hand of ahuman body.

Thus, when electromagnetic waves or energy signals are transmitted by atransmitter, the returning waves received by a receiver provide anindication of the absorption of the electromagnetic energy. That is,absorption of electromagnetic energy will vary depending on the presenceor absence of a human occupant, the occupant's size, bulk, surfacereflectivity, etc. depending on the frequency, so that different signalswill be received relating to the degree or extent of absorption by theoccupying item on the seat. The receiver will produce a signalrepresentative of the returned waves or energy signals which will thusconstitute an absorption signal as it corresponds to the absorption ofelectromagnetic energy by the occupying item in the seat.

One or more of the transducers 6, 8, 10 can also be image-receivingdevices, such as cameras, which take images of the interior of thepassenger compartment. These images can be transmitted to a remotefacility to monitor the passenger compartment or can be stored in amemory device for use in the event of an accident, i.e., to determinethe status of the occupant(s) of the vehicle prior to the accident. Inthis manner, it can be ascertained whether the driver was fallingasleep, talking on the phone, etc.

A memory device for storing images of the passenger compartment, andalso for receiving and storing any other information, parameters andvariables relating to the vehicle or occupancy of the vehicle, may be inthe form a standardized “black box” (instead of or in addition to amemory part in a processor 20). The IEEE Standards Association iscurrently beginning to develop an international standard for motorvehicle event data recorders. The information stored in the black boxand/or memory unit in the processor 20, can include the images of theinterior of the passenger compartment as well as the number of occupantsand the health state of the occupant(s). The black box would preferablybe tamper-proof and crash-proof and enable retrieval of the informationafter a crash.

Transducer 8 can also be a source of electromagnetic radiation, such asan LED, and transducers 6 and 10 can be CMOS, CCD imagers or otherdevices sensitive to electromagnetic radiation or fields. This “image”or return signal will differ for each object that is placed on thevehicle seat, or elsewhere in the vehicle, and it will also change foreach position of a particular object and for each position of thevehicle seat or other movable objects within the vehicle. Elements 6, 8,10, although described as transducers, are representative of any type ofcomponent used in a wave-based or electric field analysis technique,including, e.g., a transmitter, receiver, antenna or a capacitor plate.

Transducers 12, 14 and 16 can be antennas placed in the seat andinstrument panel, or other convenient location within the vehicle, suchthat the presence of an object, particularly a water-containing objectsuch as a human, disturbs the near field of the antenna. Thisdisturbance can be detected by various means such as with Micrel partsMICREF102 and MICREF104, which have a built-in antenna auto-tunecircuit. Note, these parts cannot be used as is and it is necessary toredesign the chips to allow the auto-tune information to be retrievedfrom the chip.

Other types of transducers can be used along with the transducers 6, 8,10 or separately and all are contemplated by at least one of theinventions disclosed herein. Such transducers include other wave devicessuch as radar or electronic field sensing systems such as described inU.S. Pat. No. 5,366,241, U.S. Pat. No. 5,602,734, U.S. Pat. No.5,691,693, U.S. Pat. No. 5,802,479, U.S. Pat. No. 5,844,486, U.S. Pat.No. 6,014,602, and U.S. Pat. No. 6,275,146 to Kithil, and U.S. Pat. No.5,948,031 to Rittmueller. Another technology, for example, uses the factthat the content of the near field of an antenna affects the resonanttuning of the antenna. Examples of such a device are shown as antennas12, 14 and 16 in FIG. 1. By going to lower frequencies, the near fieldrange is increased and also at such lower frequencies, a ferrite-typeantenna could be used to minimize the size of the antenna. Otherantennas that may be applicable for a particular implementation includedipole, microstrip, patch, Yagi etc. The frequency transmitted by theantenna can be swept and the (VSWR) voltage and current in the antennafeed circuit can be measured. Classification by frequency domain is thenpossible. That is, if the circuit is tuned by the antenna, the frequencycan be measured to determine the object in the field.

An alternate system is shown in FIG. 2, which is a side view showingschematically the interface between the vehicle interior monitoringsystem of at least one of the inventions disclosed herein and thevehicle cellular or other communication system 32, such as a satellitebased system such as that supplied by Skybitz, having an associatedantenna 34. In this view, an adult occupant 30 is shown sitting on thefront passenger seat 4 and two transducers 6 and 8 are used to determinethe presence (or absence) of the occupant on that seat 4. One of thetransducers 8 in this case acts as both a transmitter and receiver whilethe other transducer 6 acts only as a receiver. Alternately, transducer6 could serve as both a transmitter and receiver or the transmittingfunction could be alternated between the two devices. Also, in manycases, more that two transmitters and receivers are used and in stillother cases, other types of sensors, such as weight, chemical,radiation, vibration, acoustic, seatbelt tension sensor or switch,heartbeat, self tuning antennas (12, 14), motion and seat and seatbackposition sensors, are also used alone or in combination with thetransducers 6 and 8. As is also the case in FIG. 1, the transducers 6and 8 are attached to the vehicle embedded in the A-pillar and headlinertrim, where their presence is disguised, and are connected to processor20 that may also be hidden in the trim as shown or elsewhere. Othermounting locations can also be used and, in most cases, preferred asdisclosed in U.S. RE 37260.

The transducers 6 and 8 in conjunction with the pattern recognitionhardware and software described below enable the determination of thepresence of an occupant within a short time after the vehicle isstarted. The software is implemented in processor 20 and is packaged ona printed circuit board or flex circuit along with the transducers 6 and8. Similar systems can be located to monitor the remaining seats in thevehicle, also determine the presence of occupants at the other seatinglocations and this result is stored in the computer memory, which ispart of each monitoring system processor 20. Processor 20 thus enables acount of the number of occupants in the vehicle to be obtained byaddition of the determined presence of occupants by the transducersassociated with each seating location, and in fact, can be designed toperform such an addition, the principles illustrated for automobilevehicles are applicable by those skilled in the art to other vehiclessuch as shipping containers or truck trailers and to other compartmentsof an automotive vehicle such as the vehicle trunk.

For a general object, transducers 6, 8, 9, 10 can also be used todetermine the type of object, determine the location of the object,and/or determine another property or characteristic of the object. Aproperty of the object could be the orientation of a child seat, thevelocity of an adult and the like. For example, the transducers 6, 8, 9,10 can be designed to enable a determination that an object is presenton the seat, that the object is a child seat and that the child seat isrear-facing.

The transducers 6 and 8 are attached to the vehicle buried in the trimsuch as the A-pillar trim, where their presence can be disguised, andare connected to processor 20 that may also be hidden in the trim asshown (this being a non-limiting position for the processor 20). TheA-pillar is the roof support pillar that is closest to the front of thevehicle and which, in addition to supporting the roof, also supports thefront windshield and the front door. Other mounting locations can alsobe used. For example, transducers 6, 8 can be mounted inside the seat(along with or in place of transducers 12 and 14), in the ceiling of thevehicle, in the B-pillar, in the C-pillar and in the doors. Indeed, thevehicle interior monitoring system in accordance with the invention maycomprise a plurality of monitoring units, each arranged to monitor aparticular seating location. In this case, for the rear seatinglocations, transducers might be mounted in the B-pillar or C-pillar orin the rear of the front seat or in the rear side doors. Possiblemounting locations for transducers, transmitters, receivers and otheroccupant sensing devices are disclosed in above-referenced patentapplications and all of these mounting locations are contemplated foruse with the transducers described herein.

The cellular phone or other communications system 32 outputs to anantenna 34. The transducers 6, 8, 12 and 14 in conjunction with thepattern recognition hardware and software, which is implemented inprocessor 20 and is packaged on a printed circuit board or flex circuitalong with the transducers 6 and 8, determine the presence of anoccupant within a few seconds after the vehicle is started, or within afew seconds after the door is closed. Similar systems located to monitorthe remaining seats in the vehicle, also determine the presence ofoccupants at the other seating locations and this result is stored inthe computer memory which is part of each monitoring system processor20.

Periodically and in particular in the event of an accident, theelectronic system associated with the cellular phone system 32interrogates the various interior monitoring system memories and arrivesat a count of the number of occupants in the vehicle, and optionally,even makes a determination as to whether each occupant was wearing aseatbelt and if he or she is moving after the accident. The phone orother communications system then automatically dials the EMS operator(such as 911 or through a telematics service such as OnStar®) and theinformation obtained from the interior monitoring systems is forwardedso that a determination can be made as to the number of ambulances andother equipment to send to the accident site, for example. Such vehicleswill also have a system, such as the global positioning system, whichpermits the vehicle to determine its exact location and to forward thisinformation to the EMS operator. Other systems can be implemented inconjunction with the communication with the emergency services operator.For example, a microphone and speaker can be activated to permit theoperator to attempt to communicate with the vehicle occupant(s) andthereby learn directly of the status and seriousness of the condition ofthe occupant(s) after the accident.

Thus, in basic embodiments of the invention, wave or otherenergy-receiving transducers are arranged in the vehicle at appropriatelocations, trained if necessary depending on the particular embodiment,and function to determine whether a life form is present in the vehicleand if so, how many life forms are present and where they are locatedetc. To this end, transducers can be arranged to be operative at only asingle seating location or at multiple seating locations with aprovision being made to eliminate a repetitive count of occupants. Adetermination can also be made using the transducers as to whether thelife forms are humans, or more specifically, adults, child in childseats, etc. As noted herein, this is possible using pattern recognitiontechniques. Moreover, the processor or processors associated with thetransducers can be trained to determine the location of the life forms,either periodically or continuously or possibly only immediately before,during and after a crash. The location of the life forms can be asgeneral or as specific as necessary depending on the systemrequirements, i.e., a determination can be made that a human is situatedon the driver's seat in a normal position (general) or a determinationcan be made that a human is situated on the driver's seat and is leaningforward and/or to the side at a specific angle as well as the positionof his or her extremities and head and chest (specifically). The degreeof detail is limited by several factors, including, for example, thenumber and position of transducers and training of the patternrecognition algorithm(s).

In addition to the use of transducers to determine the presence andlocation of occupants in a vehicle, other sensors could also be used.For example, a heartbeat sensor which determines the number and presenceof heartbeat signals can also be arranged in the vehicle, which wouldthus also determine the number of occupants as the number of occupantswould be equal to the number of heartbeat signals detected. Conventionalheartbeat sensors can be adapted to differentiate between a heartbeat ofan adult, a heartbeat of a child and a heartbeat of an animal. As itsname implies, a heartbeat sensor detects a heartbeat, and the magnitudeand/or frequency thereof, of a human occupant of the seat, if such ahuman occupant is present. The output of the heartbeat sensor is inputto the processor of the interior monitoring system. One heartbeat sensorfor use in the invention may be of the types as disclosed in McEwan(U.S. Pat. Nos. 5,573,012 and 5,766,208). The heartbeat sensor can bepositioned at any convenient position relative to the seats whereoccupancy is being monitored. A preferred location is within the vehicleseatback.

An alternative way to determine the number of occupants is to monitorthe weight being applied to the seats, i.e., each seating location, byarranging weight sensors at each seating location which might also beable to provide a weight distribution of an object on the seat. Analysisof the weight and/or weight distribution by a predetermined method canprovide an indication of occupancy by a human, an adult or child, or aninanimate object.

Another type of sensor which is not believed to have been used in aninterior monitoring system previously is a micropower impulse radar(MIR) sensor which determines motion of an occupant and thus candetermine his or her heartbeat (as evidenced by motion of the chest).Such an MIR sensor can be arranged to detect motion in a particular areain which the occupant's chest would most likely be situated or could becoupled to an arrangement which determines the location of theoccupant's chest and then adjusts the operational field of the MIRsensor based on the determined location of the occupant's chest. Amotion sensor utilizing a micro-power impulse radar (MIR) system asdisclosed, for example, in McEwan (U.S. Pat. No. 5,361,070), as well asmany other patents by the same inventor.

Motion sensing is accomplished by monitoring a particular range from thesensor as disclosed in that patent. MIR is one form of radar which hasapplicability to occupant sensing and can be mounted at variouslocations in the vehicle. It has an advantage over ultrasonic sensors inthat data can be acquired at a higher speed and thus the motion of anoccupant can be more easily tracked. The ability to obtain returns overthe entire occupancy range is somewhat more difficult than withultrasound resulting in a more expensive system overall. MIR hasadditional advantages in lack of sensitivity to temperature variationand has a comparable resolution to about 40 kHz ultrasound. Resolutioncomparable to higher frequency ultrasound is also possible.Additionally, multiple MIR sensors can be used when high speed trackingof the motion of an occupant during a crash is required since they canbe individually pulsed without interfering with each through timedivision multiplexing.

An alternative way to determine motion of the occupant(s) is to monitorthe weight distribution of the occupant whereby changes in weightdistribution after an accident would be highly suggestive of movement ofthe occupant. A system for determining the weight distribution of theoccupants could be integrated or otherwise arranged in the seats such asthe front seat 4 of the vehicle and several patents and publicationsdescribe such systems.

More generally, any sensor which determines the presence and healthstate of an occupant can also be integrated into the vehicle interiormonitoring system in accordance with the invention. For example, asensitive motion sensor can determine whether an occupant is breathingand a chemical sensor can determine the amount of carbon dioxide, or theconcentration of carbon dioxide, in the air in the passenger compartmentof the vehicle which can be correlated to the health state of theoccupant(s). The motion sensor and chemical sensor can be designed tohave a fixed operational field situated where the occupant's mouth ismost likely to be located. In this manner, detection of carbon dioxidein the fixed operational field could be used as an indication of thepresence of a human occupant in order to enable the determination of thenumber of occupants in the vehicle. In the alternative, the motionsensor and chemical sensor can be adjustable and adapted to adjust theiroperational field in conjunction with a determination by an occupantposition and location sensor which would determine the location ofspecific parts of the occupant's body, e.g., his or her chest or mouth.Furthermore, an occupant position and location sensor can be used todetermine the location of the occupant's eyes and determine whether theoccupant is conscious, i.e., whether his or her eyes are open or closedor moving.

The use of chemical sensors can also be used to detect whether there isblood present in the vehicle, for example, after an accident.Additionally, microphones can detect whether there is noise in thevehicle caused by groaning, yelling, etc., and transmit any such noisethrough the cellular or other communication connection to a remotelistening facility (such as operated by OnStar®).

In FIG. 3, a view of the system of FIG. 1 is illustrated with a box 28shown on the front passenger seat in place of a rear facing child seat.The vehicle interior monitoring system is trained to recognize that thisbox 28 is neither a rear facing child seat nor an occupant and thereforeit is treated as an empty seat and the deployment of the airbag or otheroccupant restraint device is suppressed. For other vehicles, it may bethat just the presence of a box or its motion or chemical or radiationeffluents that are desired to be monitored. The auto-tune antenna-basedsystem 12, 14 is particularly adept at making this distinctionparticularly if the box 28 does not contain substantial amounts ofwater. Although a simple implementation of the auto-tune antenna systemis illustrated, it is of course possible to use multiple antennaslocated in the seat 4 and elsewhere in the passenger compartment andthese antenna systems can either operate at one or a multiple ofdifferent frequencies to discriminate type, location and/or relativesize of the object being investigated. This training can be accomplishedusing a neural network or modular neural network with the commerciallyavailable software. The system assesses the probability that the box 28is a person, however, and if there is even the remotest chance that itis a person, the airbag deployment is not suppressed. The system is thustypically biased toward enabling airbag deployment.

In cases where different levels of airbag inflation are possible, andthere are different levels of injury associated with an out of positionoccupant being subjected to varying levels of airbag deployment, it issometimes possible to permit a depowered or low level airbag deploymentin cases of uncertainty. If, for example, the neural network has aproblem distinguishing whether a box or a forward facing child seat ispresent on the vehicle seat, the decision can be made to deploy theairbag in a depowered or low level deployment state. Other situationswhere such a decision could be made would be when there is confusion asto whether a forward facing human is in position or out-of-position.

Neural networks systems frequently have problems in accuratelydiscriminating the exact location of an occupant especially whendifferent-sized occupants are considered. This results in a gray zonearound the border of the keep out zone where the system provides a weakfire or weak no fire decision. For those cases, deployment of the airbagin a depowered state can resolve the situation since an occupant in agray zone around the keep out zone boundary would be unlikely to beinjured by such a depowered deployment while significant airbagprotection is still being supplied.

Electromagnetic or ultrasonic energy can be transmitted in three modesin determining the position of an occupant, for example. In most of thecases disclosed above, it is assumed that the energy will be transmittedin a broad diverging beam which interacts with a substantial portion ofthe occupant or other object to be monitored. This method can have thedisadvantage that it will reflect first off the nearest object and,especially if that object is close to the transmitter, it may mask thetrue position of the occupant or object. It can also reflect off manyparts of the object where the reflections can be separated in time andprocessed as in an ultrasonic occupant sensing system. This can also bepartially overcome through the use of the second mode which uses anarrow beam. In this case, several narrow beams are used. These beamsare aimed in different directions toward the occupant from a positionsufficiently away from the occupant or object such that interference isunlikely.

A single receptor could be used provided the beams are either cycled onat different times or are of different frequencies. Another approach isto use a single beam emanating from a location which has an unimpededview of the occupant or object such as the windshield header in the caseof an automobile or near the roof at one end of a trailer or shippingcontainer, for example. If two spaced apart CCD array receivers areused, the angle of the reflected beam can be determined and the locationof the occupant can be calculated. The third mode is to use a singlebeam in a manner so that it scans back and forth and/or up and down, orin some other pattern, across the occupant, object or the space ingeneral. In this manner, an image of the occupant or object can beobtained using a single receptor and pattern recognition software can beused to locate the head or chest of the occupant or size of the object,for example. The beam approach is most applicable to electromagneticenergy but high frequency ultrasound can also be formed into a narrowbeam.

A similar effect to modifying the wave transmission mode can also beobtained by varying the characteristics of the receptors. Throughappropriate lenses or reflectors, receptors can be made to be mostsensitive to radiation emitted from a particular direction. In thismanner, a single broad beam transmitter can be used coupled with anarray of focused receivers, or a scanning receiver, to obtain a roughimage of the occupant or occupying object.

Each of these methods of transmission or reception could be used, forexample, at any of the preferred mounting locations shown in FIG. 5.

As shown in FIG. 2, there are provided four sets of wave-receivingsensor systems 6, 8, 9, 10 mounted within the passenger compartment ofan automotive vehicle. Each set of sensor systems 6, 8, 9, 10 comprisesa transmitter and a receiver (or just a receiver in some cases), whichmay be integrated into a single unit or individual components separatedfrom one another. In this embodiment, the sensor system 6 is mounted onthe A-Pillar of the vehicle. The sensor system 9 is mounted on the upperportion of the B-Pillar. The sensor system 8 is mounted on the roofceiling portion or the headliner. The sensor system 10 is mounted nearthe middle of an instrument panel 17 in front of the driver's seat 3.

The sensor systems 6, 8, 9, 10 are preferably ultrasonic orelectromagnetic, although sensor systems 6, 8, 9, 10 can be any othertype of sensors which will detect the presence of an occupant from adistance including capacitive or electric field sensors. Also, if thesensor systems 6, 8, 9, 10 are passive infrared sensors, for example,then they may only comprise a wave-receiver. Recent advances in QuantumWell Infrared Photodetectors by NASA show great promise for thisapplication. See “Many Applications Possible For Largest QuantumInfrared Detector”, Goddard Space Center News Release Feb. 27, 2002.

The Quantum Well Infrared Photodetector is a new detector which promisesto be a low-cost alternative to conventional infrared detectortechnology for a wide range of scientific and commercial applications,and particularly for sensing inside and outside of a vehicle. The mainproblem that needs to be solved is that it operates at 76 degrees Kelvin(−323 degrees F.). Chips are being developed capable of cooling otherchips economically. It remains to be seen if these low temperatures canbe economically achieved.

A section of the passenger compartment of an automobile is showngenerally as 40 in FIGS. 8A-8D. A driver 30 of the vehicle sits on aseat 3 behind a steering wheel 42, which contains an airbag assembly 44.Airbag assembly 44 may be integrated into the steering wheel assembly orcoupled to the steering wheel 42. Five transmitter and/or receiverassemblies 49, 50, 51, 52 and 54 are positioned at various places in thepassenger compartment to determine the location of various parts of thedriver, e.g., the head, chest and torso, relative to the airbag and tootherwise monitor the interior of the passenger compartment. Monitoringof the interior of the passenger compartment can entail detecting thepresence or absence of the driver and passengers, differentiatingbetween animate and inanimate objects, detecting the presence ofoccupied or unoccupied child seats, rear-facing or forward-facing, andidentifying and ascertaining the identity of the occupying items in thepassenger compartment, a similar system can be used for monitoring theinterior of a truck, shipping container or other containers.

A processor such as control circuitry 20 is connected to thetransmitter/receiver assemblies 49, 50, 51, 52, 54 and controls thetransmission from the transmitters, if a transmission component ispresent in the assemblies, and captures the return signals from thereceivers, if a receiver component is present in the assemblies. Controlcircuitry 20 usually contains analog to digital converters (ADCs) or aframe grabber or equivalent, a microprocessor containing sufficientmemory and appropriate software including, for example, patternrecognition algorithms, and other appropriate drivers, signalconditioners, signal generators, etc. Usually, in any givenimplementation, only three or four of the transmitter/receiverassemblies would be used depending on their mounting locations asdescribed below. In some special cases, such as for a simpleclassification system, only a single or sometimes only twotransmitter/receiver assemblies are used.

A portion of the connection between the transmitter/receiver assemblies49, 50, 51, 52, 54 and the control circuitry 20, is shown as wires.These connections can be wires, either individual wires leading from thecontrol circuitry 20 to each of the transmitter/receiver assemblies 49,50, 51, 52, 54 or one or more wire buses or in some cases, wireless datatransmission can be used.

The location of the control circuitry 20 in the dashboard of the vehicleis for illustration purposes only and does not limit the location of thecontrol circuitry 20. Rather, the control circuitry 20 may be locatedanywhere convenient or desired in the vehicle.

It is contemplated that a system and method in accordance with theinvention can include a single transmitter and multiple receivers, eachat a different location. Thus, each receiver would not be associatedwith a transmitter forming transmitter/receiver assemblies. Rather, forexample, with reference to FIG. 8A, only element 51 could constitute atransmitter/receiver assembly and elements 49, 50, 52 and 54 could bereceivers only.

On the other hand, it is conceivable that in some implementations, asystem and method in accordance with the invention include a singlereceiver and multiple transmitters. Thus, each transmitter would not beassociated with a receiver forming transmitter/receiver assemblies.Rather, for example, with reference to FIG. 8A, only element 51 wouldconstitute a transmitter/receiver assembly and elements 49, 50, 52, 54would be transmitters only.

One ultrasonic transmitter/receiver as used herein is similar to thatused on modern auto-focus cameras such as manufactured by the PolaroidCorporation. Other camera auto-focusing systems use differenttechnologies, which are also applicable here, to achieve the samedistance to object determination. One camera system manufactured by Fujiof Japan, for example, uses a stereoscopic system which could also beused to determine the position of a vehicle occupant providing there issufficient light available. In the case of insufficient light, a sourceof infrared light can be added to illuminate the driver. In a relatedimplementation, a source of infrared light is reflected off of thewindshield and illuminates the vehicle occupant. An infrared receiver 56is located attached to the rear view mirror assembly 55, as shown inFIG. 8E. Alternately, the infrared can be sent by the device 50 andreceived by a receiver elsewhere. Since any of the devices shown inthese figures could be either transmitters or receivers or both, forsimplicity, only the transmitted and not the reflected wave fronts arefrequently illustrated.

When using the surface of the windshield as a reflector of infraredradiation (for transmitter/receiver assembly and element 52), care mustbe taken to assure that the desired reflectivity at the frequency ofinterest is achieved. Mirror materials, such as metals and other specialmaterials manufactured by Eastman Kodak, have a reflectivity forinfrared frequencies that is substantially higher than at visiblefrequencies. They are thus candidates for coatings to be placed on thewindshield surfaces for this purpose.

There are two preferred methods of implementing the vehicle interiormonitoring system of at least one of the inventions disclosed herein, amicroprocessor system and an application specific integrated circuitsystem (ASIC). Both of these systems are represented schematically as 20herein. In some systems, both a microprocessor and an ASIC are used. Inother systems, most if not all of the circuitry is combined onto asingle chip (system on a chip). The particular implementation depends onthe quantity to be made and economic considerations. A block diagramillustrating the microprocessor system is shown in FIG. 9A which showsthe implementation of the system of FIG. 1. An alternate implementationof the FIG. 1 system using an ASIC is shown in FIG. 9B. In both cases,the target, which may be a rear facing child seat, is shownschematically as 2 and the three transducers as 6, 8, and 10. In theembodiment of FIG. 9A, there is a digitizer coupled to the receivers 6,10 and the processor, and an indicator coupled to the processor. In theembodiment of FIG. 9B, there is a memory unit associated with the ASICand also an indicator coupled to the ASIC.

The position of the occupant may be determined in various ways includingby receiving and analyzing waves from a space in a passenger compartmentof the vehicle occupied by the occupant, transmitting waves to impactthe occupant, receiving waves after impact with the occupant andmeasuring time between transmission and reception of the waves,obtaining two or three-dimensional images of a passenger compartment ofthe vehicle occupied by the occupant and analyzing the images with anoptional focusing of the images prior to analysis, or by moving a beamof radiation through a passenger compartment of the vehicle occupied bythe occupant. The waves may be ultrasonic, radar, electromagnetic,passive infrared, and the like, and capacitive in nature. In the lattercase, a capacitance or capacitive sensor may be provided. An electricfield sensor could also be used.

Deployment of the airbag can be disabled when the determined position istoo close to the airbag.

The rate at which the airbag is inflated and/or the time in which theairbag is inflated may be determined based on the determined position ofthe occupant.

A system for controlling deployment of an airbag comprises a determiningsystem for determining the position of an occupant to be protected bydeployment of the airbag, a sensor system for assessing the probabilitythat a crash requiring deployment of the airbag is occurring, and acircuit coupled to the determining system, the sensor system and theairbag for enabling deployment of the airbag in consideration of thedetermined position of the occupant and the assessed probability that acrash is occurring. The circuit is structured and arranged to analyzethe assessed probability relative to a pre-determined threshold wherebydeployment of the airbag is enabled only when the assessed probabilityis greater than the threshold. Further, the circuit are arranged toadjust the threshold based on the determined position of the occupant.The determining system may any of the determining systems discussedabove.

One method for controlling deployment of an airbag comprises a crashsensor for providing information on a crash involving the vehicle, aposition determining arrangement for determining the position of anoccupant to be protected by deployment of the airbag and a circuitcoupled to the airbag, the crash sensor and the position determiningarrangement and arranged to issue a deployment signal to the airbag tocause deployment of the airbag. The circuit is arranged to consider adeployment threshold which varies based on the determined position ofthe occupant. Further, the circuit is arranged to assess the probabilitythat a crash requiring deployment of the airbag is occurring and analyzethe assessed probability relative to the threshold whereby deployment ofthe airbag is enabled only when the assessed probability is greater thanthe threshold.

In another implementation, the sensor algorithm may determine the ratethat gas is generated to affect the rate at which the airbag isinflated. In all of these cases the position of the occupant is used toaffect the deployment of the airbag either as to whether or not itshould be deployed at all, the time of deployment or as to the rate ofinflation.

1.1 Optics

In FIG. 4, the ultrasonic transducers of the previous designs arereplaced by laser transducers 8 and 9 which are connected to amicroprocessor 20. In all other manners, the system operates the same.The design of the electronic circuits for this laser system is describedin U.S. Pat. No. 5,653,462 and in particular FIG. 8 thereof and thecorresponding description. In this case, a pattern recognition systemsuch as a neural network system is employed and uses the demodulatedsignals from the laser transducers 8 and 9.

A more complicated and sophisticated system is shown conceptually inFIG. 5 where transmitter/receiver assembly 52 is illustrated. In thiscase, as described briefly above, an infrared transmitter and a pair ofoptical receivers are used to capture the reflection of the passenger.When this system is used to monitor the driver as shown in FIG. 5, withappropriate circuitry and a microprocessor, the behavior of the drivercan be monitored. Using this system, not only can the position andvelocity of the driver be determined and used in conjunction with anairbag system, but it is also possible to determine whether the driveris falling asleep or exhibiting other potentially dangerous behavior bycomparing portions of his/her image over time. In this case, the speedof the vehicle can be reduced or the vehicle even stopped if this actionis considered appropriate. This implementation has the highestprobability of an unimpeded view of the driver since he/she must have aclear view through the windshield in order to operate the motor vehicle.

The output of microprocessor 20 of the monitoring system is shownconnected schematically to a general interface 36 which can be thevehicle ignition enabling system; the entertainment system; the seat,mirror, suspension or other adjustment systems; telematics or any otherappropriate vehicle system.

FIG. 8A illustrates a typical wave pattern of transmitted infrared wavesfrom transmitter/receiver assembly 49, which is mounted on the side ofthe vehicle passenger compartment above the front, driver's side door.Transmitter/receiver assembly 51, shown overlaid ontotransmitter/receiver 49, is actually mounted in the center headliner ofthe passenger compartment (and thus between the driver's seat and thefront passenger seat), near the dome light, and is aimed toward thedriver. Typically, there will be a symmetrical installation for thepassenger side of the vehicle. That is, a transmitter/receiver assemblywould be arranged above the front, passenger side door and anothertransmitter/receiver assembly would be arranged in the center headliner,near the dome light, and aimed toward the front, passenger side door.Additional transducers can be mounted in similar places for monitoringboth rear seat positions, another can be used for monitoring the trunkor any other interior volumes. As with the ultrasonic installations,most of the examples below are for automobile applications since theseare generally the most complicated. Nevertheless, at least one of theinventions disclosed herein is not limited to automobile vehicles andsimilar but generally simpler designs apply to other vehicles such asshipping containers, railroad cars and truck trailers.

In a preferred embodiment, each transmitter/receiver assembly 49, 51comprises an optical transducer, which may be a camera and an LED, thatwill frequently be used in conjunction with other opticaltransmitter/receiver assemblies such as shown at 50, 52 and 54, whichact in a similar manner. In some cases, especially when a low costsystem is used primarily to categorize the seat occupancy, a single ordual camera installation is used. In many cases, the source ofillumination is not co-located with the camera. For example, in onepreferred implementation, two cameras such as 49 and 51 are used with asingle illumination source located at 49.

These optical transmitter/receiver assemblies frequently comprise anoptical transmitter, which may be an infrared LED (or possibly a nearinfrared (NIR) LED), a laser with a diverging lens or a scanning laserassembly, and a receiver such as a CCD or CMOS array and particularly anactive pixel CMOS camera or array or a HDRL or HDRC camera or array asdiscussed below. The transducer assemblies map the location of theoccupant(s), objects and features thereof, in a two or three-dimensionalimage as will now be described in more detail.

Optical transducers using CCD arrays are now becoming price competitiveand, as mentioned above, will soon be the technology of choice forinterior vehicle monitoring. A single CCD array of 160 by 160 pixels,for example, coupled with the appropriate trained pattern recognitionsoftware, can be used to form an image of the head of an occupant andaccurately locate the head, eyes, ears etc. for some of the purposes ofat least one of the inventions disclosed herein.

The location or position of the occupant can be determined in variousways as noted and listed above and below as well. Generally, any type ofoccupant sensor can be used. Some particular occupant sensors which canbe used in the systems and methods in accordance with the invention.Specifically, a camera or other device for obtaining images of apassenger compartment of the vehicle occupied by the occupant andanalyzing the images can be mounted at the locations of the transmitterand/or receiver assemblies 49, 50, 51, and 54 in FIG. 8C. The camera orother device may be constructed to obtain three-dimensional imagesand/or focus the images on one or more optical arrays such as CCDs.Further, a mechanism for moving a beam of radiation through a passengercompartment of the vehicle occupied by the occupant, i.e., a scanningsystem, can be used. When using ultrasonic or electromagnetic waves, thetime of flight between the transmission and reception of the waves canbe used to determine the position of the occupant. The occupant sensorcan also be arranged to receive infrared radiation from a space in apassenger compartment of the vehicle occupied by the occupant. It canalso comprise an electric field sensor operative in a seat occupied bythe occupant or a capacitance sensor operative in a seat occupied by theoccupant. The implementation of such sensors in the invention will bereadily appreciated by one skilled in the art in view of the disclosureherein of general occupant sensors for sensing the position of theoccupant using waves, energy or radiation.

Looking now at FIG. 16, a schematic illustration of a system forcontrolling operation of a vehicle based on recognition of an authorizedindividual in accordance with the invention is shown. One or more imagesof the passenger compartment 105 are received at 106 and data derivedtherefrom at 107. Multiple image receivers may be provided at differentlocations. The data derivation may entail any one or more of numeroustypes of image processing techniques such as those described in U.S.Pat. No. 6,397,136 including those designed to improve the clarity ofthe image. A pattern recognition algorithm, e.g., a neural network, istrained in a training phase 108 to recognize authorized individuals. Thetraining phase can be conducted upon purchase of the vehicle by thedealer or by the owner after performing certain procedures provided tothe owner, e.g., entry of a security code or key. In the case of theoperator of a truck or when such an operator takes possession of atrailer or cargo container, the identity of the operator can be sent bytelematics to a central station for recording and perhaps furtherprocessing.

In the training phase for a theft prevention system, the authorizeddriver(s) would sit themselves in the driver or passenger seat andoptical images would be taken and processed to obtain the patternrecognition algorithm. A processor 109 is embodied with the patternrecognition algorithm thus trained to identify whether a person is theauthorized individual by analysis of subsequently obtained data derivedfrom optical images. The pattern recognition algorithm in processor 109outputs an indication of whether the person in the image is anauthorized individual for which the system is trained to identify. Asecurity system 110 enables operations of the vehicle when the patternrecognition algorithm provides an indication that the person is anindividual authorized to operate the vehicle and prevents operation ofthe vehicle when the pattern recognition algorithm does not provide anindication that the person is an individual authorized to operate thevehicle.

Optionally, an optical transmitting unit 111 is provided to transmitelectromagnetic energy into the passenger compartment, or other volumein the case of other vehicles, such that electromagnetic energytransmitted by the optical transmitting unit is reflected by the personand received by the optical image reception device 106.

As noted above, several different types of optical reception devices canbe used including a CCD array, a CMOS array, focal plane array (FPA),Quantum Well Infrared Photodetector (QWIP), any type of two-dimensionalimage receiver, any type of three-dimensional image receiver, an activepixel camera and an HDRC camera.

The processor 109 can be trained to determine the position of theindividuals included in the images obtained by the optical imagereception device, as well as the distance between the optical imagereception devices and the individuals.

Instead of a security system, another component in the vehicle can beaffected or controlled based on the recognition of a particularindividual. For example, the rear view mirror, seat, seat belt anchoragepoint, headrest, pedals, steering wheel, entertainment system, ridequality, air-conditioning/ventilation system can be adjusted.

FIG. 17 shows the components of the manner in which an environment ofthe vehicle, designated 100, is monitored. The environment may either bean interior environment (car, trailer, truck, shipping container,railroad car), the entire passenger compartment or only a part thereof,or an exterior environment. An active pixel camera 101 obtains images ofthe environment and provides the images or a representation thereof, ordata derived therefrom, to a processor 102. The processor 102 determinesat least one characteristic of an object in the environment based on theimages obtained by the active pixel camera 101, e.g., the presence of anobject in the environment, the type of object in the environment, theposition of an object in the environment, the motion of an object in theenvironment and the velocity of an object in the environment. Theenvironment can be any vehicle environment. Several active pixel camerascan be provided, each focusing on a different area of the environment,although some overlap is desired. Instead of an active pixel camera orarray, a single light-receiving pixel can be used in some cases.

Systems based on ultrasonics and neural networks have been verysuccessful in analyzing the seated-state of both the passenger anddriver seats of automobiles. Such systems are now going into productionfor preventing airbag deployment when a rear facing child seat or andout-of-position occupant is present. The ultrasonic systems, however,suffer from certain natural limitations that prevent system accuracyfrom getting better than about 99 percent. These limitations relate tothe fact that the wavelength of ultrasound is typically between 3 mm and8 mm. As a result, unexpected results occur which are due partially tothe interference of reflections from different surfaces. Additionally,commercially available ultrasonic transducers are tuned devices thatrequire several cycles before they transmit significant energy andsimilarly require several cycles before they effectively receive thereflected signals. This requirement has the effect of smearing theresolution of the ultrasound to the point that, for example, using aconventional 40 kHz transducer, the resolution of the system isapproximately three inches.

In contrast, the wavelength of near infrared is less than one micron andno significant interferences occur. Similarly, the system is not tunedand therefore is theoretically sensitive to a very few cycles. As aresult, resolution of the optical system is determined by the pixelspacing in the CCD or CMOS arrays. For this application, typical arrayshave been chosen to be 100 pixels by 100 pixels and therefore the spacebeing imaged can be broken up into pieces that are significantly lessthan 1 cm in size. If greater resolution is required, arrays havinglarger numbers of pixels are readily available. Another advantage ofoptical systems is that special lenses can be used to magnify thoseareas where the information is most critical and operate at reducedresolution where this is not the case. For example, the area closest tothe at-risk zone in front of the airbag can be magnified.

To summarize, although ultrasonic neural network systems are operatingwith high accuracy, they do not totally eliminate the problem of deathsand injuries caused by airbag deployments. Optical systems, on the otherhand, at little or no increase in cost, have the capability of virtually100 percent accuracy. Additional problems of ultrasonic systems arisefrom the slow speed of sound and diffraction caused by variations is airdensity. The slow sound speed limits the rate at which data can becollected and thus eliminates the possibility of tracking the motion ofan occupant during a high speed crash.

In an embodiment wherein electromagnetic energy is used, it is to beappreciated that any portion of the electromagnetic signals thatimpinges upon a body portion of the occupant is at least partiallyabsorbed by the body portion. Sometimes, this is due to the fact thatthe human body is composed primarily of water, and that electromagneticenergy at certain frequencies can be readily absorbed by water. Theamount of electromagnetic signal absorption is related to the frequencyof the signal, and size or bulk of the body portion that the signalimpinges upon. For example, a torso of a human body tends to absorb agreater percentage of electromagnetic energy as compared to a hand of ahuman body for some frequencies.

Thus, when electromagnetic waves or energy signals are transmitted by atransmitter, the returning waves received by a receiver provide anindication of the absorption of the electromagnetic energy. That is,absorption of electromagnetic energy will vary depending on the presenceor absence of a human occupant, the occupant's size, bulk, etc., so thatdifferent signals will be received relating to the degree or extent ofabsorption by the occupying item on a seat or elsewhere in the vehicle.The receiver will produce a signal representative of the returned wavesor energy signals which will thus constitute an absorption signal as itcorresponds to the absorption of electromagnetic energy by the occupyingitem in the seat.

Another optical infrared transmitter and receiver assembly is showngenerally at 52 in FIG. 5 and is mounted onto the instrument panelfacing the windshield. Although not shown in this view, reference 52consists of three devices, one transmitter and two receivers, one oneach side of the transmitter. In this case, the windshield is used toreflect the illumination light, and also the light reflected back by thedriver, in a manner similar to the “heads-up” display which is now beingoffered on several automobile models. The “heads-up” display, of course,is currently used only to display information to the driver and is notused to reflect light from the driver to a receiver. In this case, thedistance to the driver is determined stereoscopically through the use ofthe two receivers. In its most elementary sense, this system can be usedto measure the distance between the driver and the airbag module. Inmore sophisticated applications, the position of the driver, andparticularly of the driver's head, can be monitored over time and anybehavior, such as a drooping head, indicative of the driver fallingasleep or of being incapacitated by drugs, alcohol or illness can bedetected and appropriate action taken. Other forms of radiationincluding visual light, radar, terahertz and microwaves as well as highfrequency ultrasound could also be used by those skilled in the art.

A passive infrared system could be used to determine the position of anoccupant relative to an airbag or even to detect the presence of a humanor other life form in a vehicle. Passive infrared measures the infraredradiation emitted by the occupant and compares it to the background. Assuch, unless it is coupled with an imager and a pattern recognitionsystem, it can best be used to determine that an occupant is movingtoward the airbag since the amount of infrared radiation would then beincreasing. Therefore, it could be used to estimate the velocity of theoccupant but not his/her position relative to the airbag, since theabsolute amount of such radiation will depend on the occupant's size,temperature and clothes as well as on his position. When passiveinfrared is used in conjunction with another distance measuring system,such as the ultrasonic system described above, the combination would becapable of determining both the position and velocity of the occupantrelative to the airbag. Such a combination would be economical sinceonly the simplest circuits would be required. In one implementation, forexample, a group of waves from an ultrasonic transmitter could be sentto an occupant and the reflected group received by a receiver. Thedistance to the occupant would be proportional to the time between thetransmitted and received groups of waves and the velocity determinedfrom the passive infrared system. This system could be used in any ofthe locations illustrated in FIG. 5 as well as others not illustratedincluding truck trailers and cargo containers.

Recent advances in Quantum Well Infrared Photodetectors (QWIP) areparticularly applicable here due to the range of frequencies that theycan be designed to sense (3-18 microns) which encompasses the radiationnaturally emitted by the human body. Currently, QWIPs need to be cooledand thus are not quite ready for vehicle applications. There are,however, longer wave IR detectors based of focal plane arrays (FPA) thatare available in low resolution now. As the advantages of SWIR, MWIR andLWIR become more evident, devices that image in this part of theelectromagnetic spectrum will become more available.

Passive infrared could also be used effectively in conjunction with apattern recognition system. In this case, the passive infrared radiationemitted from an occupant can be focused onto a QWIP or FPA or even a CCDarray, in some cases, and analyzed with appropriate pattern recognitioncircuitry, or software, to determine the position of the occupant. Sucha system could be mounted at any of the preferred mounting locationsshown in FIG. 5 as well as others not illustrated.

Lastly, it is possible to use a modulated scanning beam of radiation anda single pixel receiver, PIN or avalanche diode, in the inventionsdescribed above. Any form of energy or radiation used above may also bein the infrared or radar spectrums and may be polarized and filters maybe used in the receiver to block out sunlight etc. These filters may benotch filters and may be made integral with the lens as one or morecoatings on the lens surface as is well known in the art. Note, in manyapplications, this may not be necessary as window glass blocks all IRexcept the near IR.

For some cases, such as a laser transceiver that may contain a CMOSarray, CCD, PIN or avalanche diode or other light sensitive devices, ascanner is also required that can be either solid state as in the caseof some radar systems based on a phased array, an acoustical opticalsystem as is used by some laser systems, or a mirror or MEMS basedreflecting scanner, or other appropriate technology.

An optical classification system using a single or dual camera designwill now be discussed, although more than two cameras can also be usedin the system described below. The occupant sensing system shouldperform occupant classification as well as position tracking since bothare critical information for making decision of airbag deployment in anauto accident. For other purposes such as container or truck trailermonitoring generally only classification is required. FIG. 18 shows apreferred occupant sensing strategy. Occupant classification may be donestatically since the type of occupant does not change frequently.Position tracking, however, has to be done dynamically so that theoccupant can be tracked reliably during pre-crash braking situations.Position tracking should provide continuous position information so thatthe speed and the acceleration of the occupant can be estimated and aprediction can be made even before the next actual measurement takesplace.

The current assignee has demonstrated that occupant classification anddynamic position tracking can be done with a stand-alone optical systemthat uses a single camera. The same image information is processed in asimilar fashion for both classification and dynamic position tracking.As shown in FIG. 19, the whole process can involve five steps: imageacquisition, image preprocessing, feature extraction, neural networkprocessing, and post-processing. These steps will now be discussed.

Step-1 image acquisition is to obtain the image from the imaginghardware. The imaging hardware main components may include one or moreof the following image acquisition devices, a digital CMOS camera, ahigh-power near-infrared LED, and the LED control circuit. A pluralityof such image acquisition devices can be used. This step also includesimage brightness detection and LED control for illumination. Note thatthe image brightness detection and LED control do not have to beperformed for every frame. For example, during a specific interval, theECU can turn the LED ON and OFF and compare the resulting images. If theimage with LED ON is significantly brighter, then it is identified asnighttime condition and the LED will remain ON; otherwise, it isidentified as daytime condition and the LED can remain OFF.

Step-2 image preprocessing performs such activities as removing randomnoise and enhancing contrast. Under daylight condition, the imagecontains unwanted contents because the background is illuminated bysunlight. For example, the movement of the driver, other passengers inthe backseat, and the scenes outside the passenger window can interfereif they are visible in the image. Usually, these unwanted contentscannot be completely eliminated by adjusting the camera position, butthey can be removed by image preprocessing. This process is much lesscomplicated for some vehicle monitoring cases such as trailer and cargocontainers where sunlight is rarely a problem.

Step-3 feature extraction compresses the data from, for example, the76,800 image pixels in the prototype camera to only a few hundredfloating-point numbers, which may be based of edge detection algorithms,while retaining most of the important information. In this step, theamount of the data is significantly reduced so that it becomes possibleto process the data using neural networks in Step-4.

There are many methods to extract information from an image for thepurposes herein. One preferred method is to extract information as tothe location of the edges of an object and then to input thisinformation into a pattern recognition algorithm. As will be discussedbelow, the location and use of the edges of an occupying item asfeatures in an imager is an important contribution of the inventionsdisclosed herein for occupant or other object sensing and tracking in avehicle.

Steps 2 and 3, image pre-processing and feature extraction can becombined or both performed separately by use of one or more of the imagesubtraction techniques described herein. One image subtraction techniqueis to compare a later obtained image from a series of images from animager to at least one previously obtained image from the series ofimages from the same imager to ascertain the presence of differencesbetween the images. Step 4, below, is the analysis of the differences toobtain information about the occupying item. There are various ways tocompare images, one of which is to subtract the later obtained imagefrom the previously obtained image to determine which image pixels havechanged in value. In one embodiment, the previously obtained image isobtained without infrared illumination and the later obtained image isobtained with infrared illumination. Comparison of the images can thusinvolve subtracting the later obtained image from the previouslyobtained image or subtracting the previously obtained image from thelater obtained image. The images may be obtained from a single imagingdevice mounted in the vehicle to obtain the entire series of images. Insome embodiments, each previously obtained image is an image of thecompartment including permanent structure in the compartment without atemporary occupying item. Comparison of images may thus involvesubtracting each previously obtained image from the later obtained imageto thereby remove the effect of the presence of permanent structure fromthe later obtained image. In one embodiment, edges of shapes in theimages are determined prior to comparison of the images such that onlythe determined edges of the shapes of the previously obtained image andthe later obtained image are compared to one another. Comparison ofimages may involve determining which reflections of the occupying itemremain static for all of the images and determining which reflectionsmove between the images whereby analysis of the differences constitutesanalyzing the reflections which move.

Step-4, to increase the system learning capability and performancestability, modular or combination neural networks can be used with eachmodule handling a different subtask (for example, to handle eitherdaytime or nighttime condition, or to classify a specific occupantgroup). In an optical embodiment for analysis of an occupying item in aseat, after use of image subtraction techniques to images, differencesbetween the images may be analyzed using a neural network or moregenerally any trained pattern recognition system, to determine, forexample, position of the occupying item such as the position relative toan occupant protection system, to determine the presence of a head orchest of a human occupant if a human is the occupying item and theposition of the head or chest, when present, relative to a deployableoccupant protection system, to determine motion of the occupying itemand to determine an identity of the occupying item. As a result of theanalysis, deployment of the occupant protection system can becontrolled, e.g., based on the position of the head or chest of theoccupant when the presence of an occupant is determined.

Step-5 post-processing removes random noise in the neural networkoutputs via filtering. Besides filtering, additional knowledge can beused to remove some of the undesired changes in the neural networkoutput. For example, it is impossible to change from an adult passengerto a child restraint without going through an empty-seat state orkey-off. After post-processing, the final decision of classification isoutput to the airbag control module, or other system, and it is up tothe automakers or vehicle owners or managers to decide how to utilizethe information. A set of display LED's on the instrument panel providesthe same information to the vehicle occupant(s).

If multiple images are acquired substantially simultaneously, each by adifferent image acquisition device, then each image can be processed inthe manner above. A comparison of the classification of the occupantobtained from the processing of the image obtained by each imageacquisition device can be performed to ascertain any variations. Ifthere are no variations, then the classification of the occupant islikely to be very accurate. However, in the presence of variations, thenthe images can be discarded and new images acquired until variations areeliminated.

A majority approach might also be used. For example, if three or moreimages are acquired by three different cameras, or other imagers, thenif two provide the same classification, this classification will beconsidered the correct classification. Alternately, all of the data fromall of the images can be analyzed and together in one combined neuralnetwork or combination neural network.

Referring again to FIG. 18, after the occupant is classified from theacquired image or images, i.e., as an empty seat (classification 1), aninfant carrier or an occupied rearward-facing child seat (classification2), a child or occupied forward-facing child seat (classification 3) oran adult passenger (classification 4), additional classification may beperformed for the purpose of determining a recommendation for control ofa vehicular component such as an occupant restraint device.

For classifications 1 and 2, the recommendation is always to suppressdeployment of the occupant restraint device. For classifications 3 and4, dynamic position tracking is performed. This involves the training ofneural networks or other pattern recognition techniques, one for eachclassification, so that once the occupant is classified, the particularneural network can be trained to analyze the dynamic position of thatoccupant will be used. That is, the data from acquired images will beinput to the neural network to determine a recommendation for control ofthe occupant restraint device and also into the neural network fordynamic position tracking of an adult passenger when the occupant isclassified as an adult passenger. The recommendation may be either asuppression of deployment, a depowered deployment or a full powerdeployment.

To additionally summarize, the system described can be a single ormultiple camera or other imager system where the cameras are typicallymounted on the roof or headliner of the vehicle either on the roof railsor center or other appropriate location. The source of illumination istypically one or more infrared LEDs and if infrared, the images aretypically monochromic, although color can effectively be used whennatural illumination is available. Images can be obtained at least asfast as 100 frames per second; however, slower rates are frequentlyadequate. A pattern recognition algorithmic system can be used toclassify the occupancy of a seat into a variety of classes such as: (1)an empty seat; (2) an infant seat which can be further classified asrear or forward facing; (3) a child which can be further classified asin or out-of-position and (4) an adult which can also be furtherclassified as in or out-of-position. Such a system can be used tosuppress the deployment of an occupant restraint. If the occupant isfurther tracked so that his or her position relative to the airbag, forexample, is known more accurately, then the airbag deployment can betailored to the position of the occupant. Such tracking can beaccomplished since the location of the head of the occupant is eitherknown from the analysis or can be inferred due to the position of otherbody parts.

As will be discussed in more detail below, data and images from theoccupant sensing system, which can include an assessment of the type andmagnitude of injuries, along with location information if available, canbe sent to an appropriate off-vehicle location such as an emergencymedical system (EMS) receiver either directly by cell phone, forexample, via a telematics system such as OnStar®, or over the internetif available in order to aid the service in providing medical assistanceand to access the urgency of the situation. The system can additionallybe used to identify that there are occupants in the vehicle that hasbeen parked, for example, and to start the vehicle engine and heater ifthe temperature drops below a safe threshold or to open a window oroperate the air conditioning in the event that the temperature raises toa temperature above a safe threshold. In both cases, a message can besent to the EMS or other services by any appropriate method such asthose listed above. A message can also be sent to the owner's beeper orPDA.

The system can also be used alone or to augment the vehicle securitysystem to alert the owner or other person or remote site that thevehicle security has been breeched so as to prevent danger to areturning owner or to prevent a theft or other criminal act. Asdiscussed herein, one method of alerting the owner or another interestedperson is through a satellite communication with a service such asSkybitz or equivalent. The advantage here is that the power required tooperate the system can be supplied by a long life battery and thus thesystem can be independent of the vehicle power system.

As discussed above and below, other occupant sensing systems can also beprovided that monitor the breathing or other motion of the driver, forexample, including the driver's heartbeat, eye blink rate, gestures,direction or gaze and provide appropriate responses including thecontrol of a vehicle component including any such components listedherein. If the driver is falling asleep, for example, a warning can beissued and eventually the vehicle directed off the road if necessary.

The combination of a camera system with a microphone and speaker allowsfor a wide variety of options for the control of vehicle components. Asophisticated algorithm can interpret a gesture, for example, that maybe in response to a question from the computer system. The driver mayindicate by a gesture that he or she wants the temperature to change andthe system can then interpret a “thumbs up” gesture for highertemperature and a “thumbs down” gesture for a lower temperature. When itis correct, the driver can signal by gesture that it is fine. A verylarge number of component control options exist that can be entirelyexecuted by the combination of voice, speakers and a camera that can seegestures. When the system does not understand, it can ask to have thegesture repeated, for example, or it can ask for a confirmation. Note,the presence of an occupant in a seat can even be confirmed by a wordspoken by the occupant, for example, which can use a technology known asvoice print if it is desired to identify the particular occupant.

It is also to be noted that the system can be trained to recognizeessentially any object or object location that a human can recognize andeven some that a human cannot recognize since the system can have thebenefit of special illumination as discussed above. If desired, aparticular situation such as the presence of a passenger's feet on theinstrument panel, hand on a window frame, head against the side window,or even lying down with his or her head in the lap of the driver, forexample, can be recognized and appropriate adjustments to a componentperformed.

Note, it has been assumed that the camera would be permanently mountedin the vehicle in the above discussion. This need not be the case andespecially for some after-market products, the camera function can besupplied by a cell phone or other device and a holder appropriately (andremovably) mounted in the vehicle.

Again the discussion above related primarily to sensing the interior ofand automotive vehicle for the purposes of controlling a vehiclecomponent such as a restraint system. When the vehicle is a shippingcontainer then different classifications can be used depending on theobjective. If it is to determine whether there is a life form movingwithin the container, a stowaway, for example, then that can be oneclassification. Another may be the size of a cargo box or whether it ismoving. Still another may be whether there is an unauthorized entry inprogress or that the door has been opened. Others include the presenceof a particular chemical vapor, radiation, excessive temperature,excessive humidity, excessive shock, excessive vibration etc.

1.1.1 Eyesafe Application

When using optics, the use of eye-safe frequencies is critical if thereis a possibility that a human occupant is in the scanning field.Currently, active IR uses the near IR range which has wavelengths below1400 nanometers. Recent developments in the SWIR range (particularlygreater than 1400 nm and more specifically in a range of 1400 nm toabout 1700 nm) use indium gallium arsenide (InGaAs) for an imager permitmuch higher power transmissions as they are below the eye safety zone(see, e.g., Martin H. Ettenberg “A Little Night Vision”, Solutions forthe Electronic Imaging Professional, March 2005, a Cygnus Publication,www.sensorsinc.com/downloads/article_Adv.Imging_(—)305.pdf).

Use of such eyesafe IR, i.e., greater than 1400 nm, to illuminate anarea being observed is significantly advantageous since much brighterillumination can be used. If images are taken in such an illuminatedarea with a camera that is only sensitive in this range, through use ofappropriate notch filter, then the effects of sunlight and otherartificial light can be removed. This makes the system much lesssensitive to sunlight effects. It also makes the system easier to recordan image (or an edge image) of an empty seat, for example, that would beinvariant to sun or other uncontrollable illumination and thus thesystem would be more robust. The edges of a seat, for example, wouldalways look the same regardless of the external illumination.

Use of a near infrared frequency such as SWIR (above 1.4 microns) may bein the form of a laser spotlight which would pass eye safetyrequirements. This laser spotlight coupled with range gating, e.g.,through use of a notch filter, permits easy segmentation of objects inthe captured image and thus the rapid classification using, for example,a modular neural network or combination neural network system.

Application of illumination in a frequency above 1400 nm can beimplemented in any of the embodiments described herein whereinillumination is or can be provided to the vehicular compartment withimages thereof being obtained subsequent to or contemporaneous with theillumination.

1.2 Ultrasonics and Optics

In some cases, a combination of an optical system such as a camera andan ultrasonic system can be used. In this case, the optical system canbe used to acquire an image providing information as to the vertical andlateral dimensions of the scene and the ultrasound can be used toprovide longitudinal information, for example.

A more accurate acoustic system for determining the distance to aparticular object, or a part thereof, in the passenger compartment isexemplified by transducers 24 in FIG. 8E. In this case, three ultrasonictransmitter/receivers 24 are shown spaced apart mounted onto theA-pillar of the vehicle. Due to the wavelength, it is difficult to get anarrow beam using ultrasonics without either using high frequencies thathave limited range or a large transducer. A commonly available 40 kHztransducer, for example, is about 1 cm. in diameter and emits a sonicwave that spreads at about a sixty-degree angle. To reduce this anglerequires making the transducer larger in diameter. An alternate solutionis to use several transducers and to phase the transmissions from thetransducers so that they arrive at the intended part of the target inphase. Reflections from the selected part of the target are thenreinforced whereas reflections from adjacent parts encounterinterference with the result that the distance to the brightest portionwithin the vicinity of interest can be determined. A low-Q transducermay be necessary for this application.

By varying the phase of transmission from the three transducers 24, thelocation of a reflection source on a curved line can be determined. Inorder to locate the reflection source in space, at least one additionaltransmitter/receiver is required which is not co-linear with the others.The waves shown in FIG. 8E coming from the three transducers 24 areactually only the portions of the waves which arrive at the desiredpoint in space together in phase. The effective direction of these wavestreams can be varied by changing the transmission phase between thethree transmitters 24.

A determination of the approximate location of a point of interest onthe occupant can be accomplished by a CCD or CMOS array and appropriateanalysis and the phasing of the ultrasonic transmitters is determined sothat the distance to the desired point can be determined.

Although the combination of ultrasonics and optics has been described,it will now be obvious to others skilled in the art that other sensortypes can be combined with either optical or ultrasonic transducersincluding weight sensors of all types as discussed below, as well aselectric field, chemical, temperature, humidity, radiation, vibration,acceleration, velocity, position, proximity, capacitance, angular rate,heartbeat, radar, other electromagnetic, and other sensors.

1.3 Other Transducers

In FIG. 4, the ultrasonic transducers of the previous designs can bereplaced by laser or other electromagnetic wave transducers ortransceivers 8 and 9, which are connected to a microprocessor 20. Asdiscussed above, these are only illustrative mounting locations and anyof the locations described herein are suitable for particulartechnologies. Also, such electromagnetic transceivers are meant toinclude the entire electromagnetic spectrum including from X-rays to lowfrequencies where sensors such as capacitive or electric field sensorsincluding so called “displacement current sensors” as discussed indetail herein, and the auto-tune antenna sensor also discussed hereinoperate.

A block diagram of an antenna based near field object detector isillustrated in FIG. 20. The circuit variables are defined as follows:

F=Frequency of operation Hz.

ψ=2*π*F radians/second

α=Phase angle between antenna voltage and antenna current.

A, k1, k2, k3, k4 are scale factors, determined by system design.

Tp1-8 are points on FIG. 20.

Tp1=k1*Sin(ωt)

Tp2=k1*Cos(ωt) Reference voltage to phase detector

Tp3=k2*Sin(ωt) drive voltage to Antenna

Tp4=k3*Cos(ω+δ) Antenna current

Tp5=k4*Cos(ωt+δ) Voltage representing Antenna current

Tp6=0.5ωt)Sin(ωT) Output of phase detector

Tp7=Absorption signal output

Tp8=Proximity signal output

In a tuned circuit, the voltage and the current are 90 degrees out ofphase with each other at the resonant frequency. The frequency sourcesupplies a signal to the phase shifter. The phase shifter outputs twosignals that are out of phase by 90 degrees at frequency F. The drive tothe antenna is the signal Tp3. The antenna can be of any suitable typesuch as dipole, patch, Yagi etc. When the signal Tp1 from the phaseshifter has sufficient power, the power amplifier may be eliminated. Theantenna current is at Tp4, which is converted into a voltage since thephase detector requires a voltage drive. The output of the phasedetector is Tp6, which is filtered and used to drive the varactor tuningdiode D1. Multiple diodes may be used in place of diode D1. The phasedetector, amplifier filter, varactor tuning diode D1 and current tovoltage converter form a closed loop servo that keeps the antennavoltage and current in a 90-degree relationship at frequency F. Thetuning loop maintains a 90-degree phase relationship between the antennavoltage and the antenna current. When an object such as a human comesnear the antenna and attempts to detune it, the phase detector sensesthe phase change and adds or subtracts capacity by changing voltage tothe varactor tuning diode D1 thereby maintaining resonance at frequencyF.

The voltage Tp8 is an indication of the capacity of a nearby object. Anobject that is near the loop and absorbs energy from it, will change theamplitude of the signal at Tp5, which is detected and outputted to Tp7.The two signals Tp7 and Tp8 are used to determine the nature of theobject near the antenna.

An object such as a human or animal with a fairly high electricalpermittivity or dielectric constant and a relatively high lossdielectric property (high loss tangent) absorbs significant energy. Thiseffect varies with the frequency used for the detection. If a human, whohas a high loss tangent is present in the detection field, then thedielectric absorption causes the value of the capacitance of the objectto change with frequency. For a human with high dielectric losses (highloss tangent), the decay with frequency will be more pronounced than forobjects that do not present this high loss tangency. Exploiting thisphenomenon makes it possible to detect the presence of an adult, child,baby, pet or other animal in the detection field.

An older method of antenna tuning used the antenna current and thevoltage across the antenna to supply the inputs to a phase detector. Ina 25 to 50 mw transmitter with a 50 ohm impedance, the current is small,it is therefore preferable to use the method described herein.

Note that the auto-tuned antenna sensor is preferably placed in thevehicle seat, headrest, floor, dashboard, headliner, or airbag modulecover for an automotive vehicle. Seat mounted examples are shown at 12,13, 14 and 15 in FIG. 4 and a floor mounted example at 11. In most othermanners, the system operates the same. The geometry of the antennasystem would differ depending on the vehicle to which it is applied andthe intended purpose. Such a system, for example, can be designed todetect the entry of a person into a container or trailer through thedoor.

1.4 Circuits

There are several preferred methods of implementing the vehicle interiormonitoring systems of at least one of the inventions disclosed hereinincluding a microprocessor, an application specific integrated circuitsystem (ASIC), a system on a chip and/or an FPGA or DSP. These systemsare represented schematically as 20 herein. In some systems, both amicroprocessor and an ASIC are used. In other systems, most if not allof the circuitry is combined onto a single chip (system on a chip). Theparticular implementation depends on the quantity to be made andeconomic considerations. It also depends on time-to-marketconsiderations where FPGA is frequently the technology of choice.

The design of the electronic circuits for a laser system is described inU.S. Pat. No. 5,653,462 and in particular FIG. 8 thereof and thecorresponding description.

2. Adaptation

Let us now consider the process of adapting a system of occupant orobject sensing transducers to a vehicle. For example, if a candidatesystem for an automobile consisting of eight transducers is considered,four ultrasonic transducers and four weight transducers, and if costconsiderations require the choice of a smaller total number oftransducers, it is a question of which of the eight transducers shouldbe eliminated. Fortunately, the neural network technology discussedbelow provides a technique for determining which of the eighttransducers is most important, which is next most important, etc. If thesix most critical transducers are chosen, that is the six transducerswhich contain or provide the most useful information as determined bythe neural network, a neural network can be trained using data fromthose six transducers and the overall accuracy of the system can bedetermined. Experience has determined, for example, that typically thereis almost no loss in accuracy by eliminating two of the eighttransducers, for example, two of the strain gage weight sensors. Aslight loss of accuracy occurs when one of the ultrasonic transducers isthen eliminated. In this manner, by the process of adaptation, the mostcost effective system can be determined from a proposed set of sensors.

This same technique can be used with the additional transducersdescribed throughout this disclosure. A transducer space can bedetermined with perhaps twenty different transducers comprised ofultrasonic, optical, electromagnetic, electric field, motion, heartbeat,weight, seat track, seatbelt payout, seatback angle and other types oftransducers depending on the particular vehicle application. The neuralnetwork can then be used in conjunction with a cost function todetermine the cost of system accuracy. In this manner, the optimumcombination of any system cost and accuracy level can be determined.

System Adaptation involves the process by which the hardwareconfiguration and the software algorithms are determined for aparticular vehicle. Each vehicle model or platform will most likely havea different hardware configuration and different algorithms. Some of thevarious aspects that make up this process are as follows:

-   -   The determination of the mounting location and aiming or        orientation of the transducers.    -   The determination of the transducer field angles or area or        volume monitored    -   The use of a combination neural network algorithm generating        program such as available from International Scientific        Research, Inc. to help generate the algorithms or other pattern        recognition algorithm generation program. (as described below)    -   The process of the collection of data in the vehicle, for        example, for neural network training purposes.    -   The method of automatic movement of the vehicle seats or other        structures or objects etc. while data is collected    -   The determination of the quantity of data to acquire and the        setups needed to achieve a high system accuracy, typically        several hundred thousand vectors or data sets.    -   The collection of data in the presence of varying environmental        conditions such as with thermal gradients.    -   The photographing of each data setup.    -   The makeup of the different databases and the use of typically        three different databases.    -   The method by which the data is biased to give higher        probabilities for, e.g., forward facing humans.    -   The automatic recording of the vehicle setup including seat,        seat back, headrest, window, visor, armrest, and other object        positions, for example, to help insure data integrity.    -   The use of a daily setup to validate that the transducer        configuration and calibration has not changed.    -   The method by which bad data is culled from the database.    -   The inclusion of the Fourier transforms and other pre-processors        of the data in the algorithm generation process if appropriate.    -   The use of multiple algorithm levels, for example, for        categorization and position.    -   The use of multiple algorithms in parallel.    -   The use of post processing filters and the particularities of        these filters.    -   The addition of fuzzy logic or other human intelligence based        rules.    -   The method by which data errors are corrected using, for        example, a neural network.    -   The use of a neural network generation program as the pattern        recognition algorithm generating system, if appropriate.    -   The use of back propagation neural networks for training.    -   The use of vector or data normalization.    -   The use of feature extraction techniques, for ultrasonic systems        for example, including:        -   The number of data points prior to a peak.        -   The normalization factor.        -   The total number of peaks.        -   The vector or data set mean or variance.    -   The use of feature extraction techniques, for optics systems for        example, including:        -   Motion.        -   Edge detection.        -   Feature detection such as the eyes, head etc.        -   Texture detection.        -   Recognizing specific features of the vehicle.        -   Line subtraction—i.e., subtracting one line of pixels from            the adjacent line with every other line illuminated. This            works primarily only with rolling shutter cameras. The            equivalent for a snapshot camera is to subtract an            artificially illuminated image from one that is illuminated            only with natural light.    -   The use of other computational intelligence systems such as        genetic algorithms    -   The use the data screening techniques.    -   The techniques used to develop stable networks including the        concepts of old and new networks.    -   The time spent or the number of iterations spent in, and method        of, arriving at stable networks.    -   The technique where a small amount of data is collected first        such as 16 sheets followed by a complete data collection        sequence.    -   The use of a cellular neural network for high speed data        collection and analysis when electromagnetic transducers are        used.    -   The use of a support vector machine.

With respect to the line subtraction technique application to featureextraction, an image is composed of multiple lines of pixels. A rollingshutter camera, among others, is capable of obtaining or deriving animage composed of multiple lines of pixels with every other lineilluminated. An image subtraction routine is performed on such an imageso that an unilluminated line of pixels is subtracted from anilluminated line or vice versa. The same technique of subtracting anilluminated line of pixels from an unilluminated line or pixels or viceversa can be applied to other cameras wherein the entire image is eitherilluminated (by artificial means) or unilluminated (with only naturallight) and subtracted from the opposite type of image.

In addition to line subtraction or image subtraction, anotherpre-processing techniques for optics is use of differential motion todiscriminate an object from its background. Analysis of multiple imagestaken from a single camera will reveal differences attributed to motionof the object with these differences being analyzable to identify theobject. The leading edge of the variation between the images is analyzedand can be compared to outlines of known shapes to determine theidentity of the object in motion (or discriminate objects). Moreover,the outline formed by the edges of the differential images can becompared to (subtracted from) another differential image to determinemotion of the occupant. The same analysis used to determine motion ofthe occupant can also be used to determine the position of the occupantrelative to, for example, an occupant protection system, or determinewhether the occupant is out-of-position for deployment of the occupantprotections system.

The process of adapting the system to the vehicle begins with a surveyof the vehicle model. Any existing sensors, such as seat positionsensors, seat back sensors, door open sensors etc., are immediatecandidates for inclusion into the system. Input from the customer willdetermine what types of sensors would be acceptable for the finalsystem. These sensors can include: seat structure-mounted weightsensors, pad-type weight sensors, pressure-type weight sensors (e.g.,bladders), seat fore and aft position sensors, seat-mounted capacitance,electric field or antenna sensors, seat vertical position sensors, seatangular position sensors, seat back position sensors, headrest positionsensors, ultrasonic occupant sensors, optical occupant sensors,capacitive sensors, electric field sensors, inductive sensors, radarsensors, vehicle velocity and acceleration sensors, shock and vibrationsensors, temperature sensors, chemical sensors, radiation sensors, brakepressure, seatbelt force, payout and buckle sensors, accelerometers,gyroscopes, etc. A candidate array of sensors is then chosen and mountedonto the vehicle. At least one of the inventions disclosed hereincontemplates final systems including any such sensors or combinations ofsuch sensors, where appropriate, for the monitoring of the interiorand/or exterior of any vehicle as the term is defined above.

The vehicle can also be instrumented so that data input by humans isminimized. Thus, the positions of the various components in the vehiclesuch as the seats, windows, sun visor, armrest, etc. are automaticallyrecorded where possible. Also, the position of the occupant while datais being taken is also recorded through a variety of techniques such asdirect ultrasonic ranging sensors, optical ranging sensors, radarranging sensors, optical tracking sensors etc., where appropriate.Special cameras can also be installed to take one or more pictures ofthe setup to correspond to each vector of data collected or at someother appropriate frequency. Herein, a vector is used to represent a setof data collected at a particular epoch or representative of theoccupant or environment of vehicle at a particular point in time.

A standard set of vehicle setups is chosen for initial trial datacollection purposes. Typically, the initial trial will consist ofbetween 20,000 and 100,000 setups, although this range is not intendedto limit the invention.

Initial digital data collection now proceeds for the trial setup matrix.The data is collected from the transducers, digitized and combined toform to a vector of input data for analysis by a pattern recognitionsystem such as a neural network program or combination neural networkprogram. This analysis should yield a training accuracy of nearly 100%.If this is not achieved, then additional sensors are added to the systemor the configuration changed and the data collection and analysisrepeated. Note, in some cases the task is sufficiently simple that aneural network is not necessary, such as the determination that atrailer is not empty.

In addition to a variety of seating states for objects in the passengercompartment, for example, the trial database can also includeenvironmental effects such as thermal gradients caused by heat lamps andthe operation of the air conditioner and heater, or where appropriatelighting variations or other environmental variations that might affectparticular transducer types. A sample of such a matrix is presented inFIGS. 82A-82H of the '881 application, with some of the variables andobjects used in the matrix being designated or described in FIGS. 76-81Dof the '881 application for automotive occupant sensing. A similarmatrix can be generated for other vehicle monitoring applications suchas cargo containers and truck trailers. After the neural network hasbeen trained on the trial database, the trial database will be scannedfor vectors that yield erroneous results (which would likely beconsidered bad data). A study of those vectors along with vectors fromassociated in time cases are compared with the photographs to determinewhether there is erroneous data present. If so, an attempt is made todetermine the cause of the erroneous data. If the cause can be found,for example if a voltage spike on the power line corrupted the data,then the vector will be removed from the database and an attempt is madeto correct the data collection process so as to remove suchdisturbances.

At this time, some of the sensors may be eliminated from the sensormatrix. This can be determined during the neural network analysis, forexample, by selectively eliminating sensor data from the analysis to seewhat the effect if any results. Caution should be exercised here,however, since once the sensors have been initially installed in thevehicle, it requires little additional expense to use all of theinstalled sensors in future data collection and analysis.

The neural network, or other pattern recognition system, that has beendeveloped in this first phase can be used during the data collection inthe next phases as an instantaneous check on the integrity of the newvectors being collected.

The next set of data to be collected when neural networks are used, forexample, is the training database. This will usually be the largestdatabase initially collected and will cover such setups as listed, forexample, in FIGS. 82A-82H of the '881 application for occupant sensing.The training database, which may contain 500,000 or more vectors, willbe used to begin training of the neural network or other patternrecognition system. In the foregoing description, a neural network willbe used for exemplary purposes with the understanding that the inventionis not limited to neural networks and that a similar process exists forother pattern recognition systems. At least one of the inventionsdisclosed herein is largely concerned with the use of patternrecognition systems for vehicle internal monitoring. The best mode is touse trained pattern recognition systems such as neural networks. Whilethis is taking place, additional data will be collected according toFIGS. 78-80 and 83, of the '881 application, of the independent andvalidation databases.

The training database is usually selected so that it uniformly coversall seated states that are known to be likely to occur in the vehicle.The independent database may be similar in makeup to the trainingdatabase or it may evolve to more closely conform to the occupancy statedistribution of the validation database. During the neural networktraining, the independent database is used to check the accuracy of theneural network and to reject a candidate neural network design if itsaccuracy, measured against the independent database, is less than thatof a previous network architecture.

Although the independent database is not actually used in the trainingof the neural network, nevertheless, it has been found that itsignificantly influences the network structure or architecture.Therefore, a third database, the validation or real world database, isused as a final accuracy check of the chosen system. It is the accuracyagainst this validation database that is considered to be the systemaccuracy. The validation database is usually composed of vectors takenfrom setups which closely correlate with vehicle occupancy in realvehicles on the roadway or wherever they are used. Initially, thetraining database is usually the largest of the three databases. As timeand resources permit, the independent database, which perhaps starts outwith 100,000 vectors, will continue to grow until it becomesapproximately the same size or even larger than the training database.The validation database, on the other hand, will typically start outwith as few as 50,000 vectors. However, as the hardware configuration isfrozen, the validation database will continuously grow until, in somecases, it actually becomes larger than the training database. This isbecause near the end of the program, vehicles will be operating onhighways, ships, railroad tracks etc. and data will be collected in realworld situations. If in the real world tests, system failures arediscovered, this can lead to additional data being taken for both thetraining and independent databases as well as the validation database.

Once a neural network, or other pattern recognition system, has beentrained or otherwise developed using all of the available data from allof the transducers, it is expected that the accuracy of the network willbe very close to 100%. It is usually not practical to use all of thetransducers that have been used in the training of the system for finalinstallation in real production vehicle models. This is primarily due tocost and complexity considerations. Usually, the automobilemanufacturer, or other customer, will have an idea of how manytransducers would be acceptable for installation in a productionvehicle. For example, the data may have been collected using 20different transducers but the customer may restrict the final selectionto 6 transducers. The next process, therefore, is to gradually eliminatetransducers to determine what is the best combination of sixtransducers, for example, to achieve the highest system accuracy.Ideally, a series of neural networks, for example, would be trainedusing all combinations of six transducers from the 20 available. Theactivity would require a prohibitively long time. Certain constraintscan be factored into the system from the beginning to start the pruningprocess. For example, it would probably not make sense to have bothoptical and ultrasonic transducers present in the same system since itwould complicate the electronics. In fact, the customer may have decidedinitially that an optical system would be too expensive and thereforewould not be considered. The inclusion of optical transducers,therefore, serves as a way of determining the loss in accuracy as afunction of cost. Various constraints, therefore, usually allow theimmediate elimination of a significant number of the initial group oftransducers. This elimination and the training on the remainingtransducers provides the resulting accuracy loss that results.

The next step is to remove each of the transducers one at a time anddetermine which sensor has the least effect on the system accuracy. Thisprocess is then repeated until the total number of transducers has beenpruned down to the number desired by the customer. At this point, theprocess is reversed to add in one at a time those transducers that wereremoved at previous stages. It has been found, for example, that asensor that appears to be unimportant during the early pruning processcan become very important later on. Such a sensor may add a small amountof information due to the presence of various other transducers. Whereasthe various other transducers, however, may yield less information thanstill other transducers and, therefore may have been removed during thepruning process. Reintroducing the sensor that was eliminated early inthe cycle therefore can have a significant effect and can change thefinal choice of transducers to make up the system.

The above method of reducing the number of transducers that make up thesystem is but one of a variety approaches which have applicability indifferent situations. In some cases, a Monte Carlo or other statisticalapproach is warranted, whereas in other cases, a design of experimentsapproach has proven to be the most successful. In many cases, anoperator conducting this activity becomes skilled and after a whileknows intuitively what set of transducers is most likely to yield thebest results. During the process it is not uncommon to run multiplecases on different computers simultaneously. Also, during this process,a database of the cost of accuracy is generated. The automobilemanufacturer, for example, may desire to have the total of 6 transducersin the final system, however, when shown the fact that the addition ofone or two additional transducers substantially increases the accuracyof the system, the manufacturer may change his mind. Similarly, theinitial number of transducers selected may be 6 but the analysis couldshow that 4 transducers give substantially the same accuracy as 6 andtherefore the other 2 can be eliminated at a cost saving.

While the pruning process is occurring, the vehicle is subjected to avariety of real world tests and would be subjected to presentations tothe customer. The real world tests are tests that are run at differentlocations than where the fundamental training took place. It has beenfound that unexpected environmental factors can influence theperformance of the system and therefore these tests can provide criticalinformation. The system therefore, which is installed in the testvehicle, should have the capability of recording system failures. Thisrecording includes the output of all of the transducers on the vehicleas well as a photograph of the vehicle setup that caused the error. Thisdata is later analyzed to determine whether the training, independent orvalidation setups need to be modified and/or whether the transducers orpositions of the transducers require modification.

Once the final set of transducers in some cases is chosen, the vehicleis again subjected to real world testing on highways, or wherever it iseventually to be used, and at customer demonstrations. Once again, anyfailures are recorded. In this case, however, since the total number oftransducers in the system is probably substantially less than theinitial set of transducers, certain failures are to be expected. Allsuch failures, if expected, are reviewed carefully with the customer tobe sure that the customer recognizes the system failure modes and isprepared to accept the system with those failure modes.

The system described so far has been based on the use of a single neuralnetwork or other pattern recognition system. It is frequently necessaryand desirable to use combination neural networks, multiple neuralnetworks, cellular neural networks or support vector machines or otherpattern recognition systems. For example, for determining the occupancystate of a vehicle seat or other part of the vehicle, there may be atleast two different requirements. The first requirement is to establishwhat is occupying the seat, for example, and the second requirement isto establish where that object is located. Another requirement might beto simply determine whether an occupying item warranting analysis by theneural networks is present. Generally, a great deal of time, typicallymany seconds, is available for determining whether a forward facinghuman or an occupied or unoccupied rear facing child seat, for example,occupies a vehicle seat. On the other hand, if the driver of the vehicleis trying to avoid an accident and is engaged in panic braking, theposition of an unbelted occupant can be changing rapidly as he or she ismoving toward the airbag. Thus, the problem of determining the locationof an occupant is time critical. Typically, the position of the occupantin such situations must be determined in less than 20 milliseconds.There is no reason for the system to have to determine that a forwardfacing human being is in the seat while simultaneously determining wherethat forward facing human being is. The system already knows that theforward facing human being is present and therefore all of the resourcescan be used to determine the occupant's position. Thus, in thissituation, a dual level or modular neural network can be advantageouslyused. The first level determines the occupancy of the vehicle seat andthe second level determines the position of that occupant. In somesituations, it has been demonstrated that multiple neural networks usedin parallel can provide some benefit. This will be discussed in moredetail below. Both modular and multiple parallel neural networks areexamples of combination neural networks.

The data fed to the pattern recognition system will usually not be theraw vectors of data as captured and digitized from the varioustransducers. Typically, a substantial amount of preprocessing of thedata is undertaken to extract the important information from the datathat is fed to the neural network. This is especially true in opticalsystems and where the quantity of data obtained, if all were used by theneural network, would require very expensive processors. The techniquesof preprocessing data will not be described in detail here. However, thepreprocessing techniques influence the neural network structure in manyways. For example, the preprocessing used to determine what is occupyinga vehicle seat is typically quite different from the preprocessing usedto determine the location of that occupant. Some particularpreprocessing concepts will be discussed in more detail below.

A pattern recognition system, such as a neural network, can sometimesmake irrational decisions. This typically happens when the patternrecognition system is presented with a data set or vector that is unlikeany vector that has been in its training set. The variety of seatingstates of a vehicle is unlimited. Every attempt is made to select fromthat unlimited universe a set of representative cases. Nevertheless,there will always be cases that are significantly different from anythat have been previously presented to the neural network. The finalstep, therefore, to adapting a system to a vehicle, is to add a measureof human intelligence or common sense. Sometimes this goes under theheading of fuzzy logic and the resulting system has been termed in somecases, a neural fuzzy system. In some cases, this takes the form of anobserver studying failures of the system and coming up with rules andthat say, for example, that if transducer A perhaps in combination withanother transducer produces values in this range, then the system shouldbe programmed to override the pattern recognition decision andsubstitute therefor a human decision.

An example of this appears in R. Scorcioni, K. Ng, M. M. Trivedi, N.Lassiter; “MoNiF: A Modular Neuro-Fuzzy Controller for Race CarNavigation”; in Proceedings of the 1997 IEEE Symposium on ComputationalIntelligence and Robotics Applications, Monterey, Calif., USA July 1997,which describes the case of where an automobile was designed forautonomous operation and trained with a neural network, in one case, anda neural fuzzy system in another case. As long as both vehicles operatedon familiar roads both vehicles performed satisfactorily. However, whenplaced on an unfamiliar road, the neural network vehicle failed whilethe neural fuzzy vehicle continued to operate successfully. If theneural network vehicle had been trained on the unfamiliar road, it mightvery well have operated successful. Nevertheless, the critical failuremode of neural networks that most concerns people is this uncertainty asto what a neural network will do when confronted with an unknown state.

One aspect, therefore, of adding human intelligence to the system, is toferret out those situations where the system is likely to fail.Unfortunately, in the current state-of-the-art, this is largely a trialand error activity. One example is that if the range of certain parts ofvector falls outside of the range experienced during training, thesystem defaults to a particular state. In the case of suppressingdeployment of one or more airbags, or other occupant protectionapparatus, this case would be to enable airbag deployment even if thepattern recognition system calls for its being disabled. An alternatemethod is to train a particular module of a modular neural network torecognize good from bad data and reject the bad data before it is fed tothe main neural networks.

The foregoing description is applicable to the systems described in thefollowing drawings and the connection between the foregoing descriptionand the systems described below will be explained below. However, itshould be appreciated that the systems shown in the drawings do notlimit the applicability of the methods or apparatus described above.

Referring again to FIG. 6, and to FIG. 6A which differs from FIG. 6 onlyin the use of a strain gage weight sensor mounted within the seatcushion, motion sensor 73 can be a discrete sensor that detects relativemotion in the passenger compartment of the vehicle. Such sensors arefrequently based on ultrasonics and can measure a change in theultrasonic pattern that occurs over a short time period. Alternately,the subtracting of one position vector from a previous position vectorto achieve a differential position vector can detect motion. For thepurposes herein, a motion sensor will be used to mean either aparticular device that is designed to detect motion for the creation ofa special vector based on vector differences or a neural network trainedto determine motion based on successive vectors.

An ultrasonic, optical or other sensor or transducer system 9 can bemounted on the upper portion of the front pillar, i.e., the A-Pillar, ofthe vehicle and a similar sensor system 6 can be mounted on the upperportion of the intermediate pillar, i.e., the B-Pillar. Each sensorsystem 6, 9 may comprise a transducer. The outputs of the sensor systems6 and 9 can be input to a band pass filter 60 through a multiplexcircuit 59 which can be switched in synchronization with a timing signalfrom the ultrasonic sensor drive circuit 58, for example, and then canbe amplified by an amplifier 61. The band pass filter 60 removes a lowfrequency wave component from the output signal and also removes some ofthe noise. The envelope wave signal can be input to an analog/digitalconverter (ADC) 62 and digitized as measured data. The measured data canbe input to a processing circuit 63, which can be controlled by thetiming signal which can be in turn output from the sensor drive circuit58. The above description applies primarily to systems based onultrasonics and will differ somewhat for optical, electric field andother systems and for different vehicle types.

Each of the measured data can be input to a normalization circuit 64 andnormalized. The normalized measured data can be input to the combinationneural network (circuit) 65, for example, as wave data.

The output of the pressure or weight sensor(s) 7, 76 or 97 (see FIG. 6A)can be amplified by an amplifier 66 coupled to the pressure or weightsensor(s) 7, 76 and 97 and the amplified output can be input to ananalog/digital converter and then directed to the neural network 65, forexample, of the processor. Amplifier 66 can be useful in someembodiments but it may be dispensed with by constructing the sensors 7,76, 97 to provide a sufficiently strong output signal, and even possiblya digital signal. One manner to do this would be to construct the sensorsystems with appropriate electronics.

The neural network 65 can be directly connected to the ADCs 68 and 69,the ADC associated with amplifier 66 and the normalization circuit 64.As such, information from each of the sensors in the system (a stream ofdata) can be passed directly to the neural network 65 for processingthereby. The streams of data from the sensors are usually not combinedprior to the neural network 65 and the neural network 65 can be designedto accept the separate streams of data (e.g., at least a part of thedata at each input node) and process them to provide an outputindicative of the current occupancy state of the seat or of the vehicle.The neural network 65 thus includes or incorporates a plurality ofalgorithms derived by training in the manners discussed herein. Once thecurrent occupancy state of the seat or vehicle is determined, it ispossible to control vehicular components or systems, such as the airbagsystem or telematics system, in consideration of the current occupancystate of the seat or vehicle.

A discussion of the methodology of adapting a monitoring system to anautomotive vehicle for the purpose primarily of controlling a componentsuch as a restraint system is described with reference to FIGS. 28-37 ofthe '934 application.

3. Mounting Locations for and Quantity of Transducers

Ultrasonic transducers are relatively good at measuring the distancealong a radius to a reflective object. An optical array, to be discussednow, on the other hand, can get accurate measurements in two dimensions,the lateral and vertical dimensions relative to the transducer. Assumingthe optical array has dimensions of 100 by 100 as compared to anultrasonic sensor that has a single dimension of 100, an optical arraycan therefore provide 100 times more information than the ultrasonicsensor. Most importantly, this vastly greater amount of information doesnot cost significantly more to obtain than the information from theultrasonic sensor.

As illustrated in FIGS. 8A-8D, the optical sensors are typically locatedfor an automotive vehicle at the positions where the desired informationis available with the greatest resolution. These positions are typicallyin the center front and center rear of the occupancy seat and at thecenter on each side and top. This is in contrast to the optimum locationfor ultrasonic sensors, which are the corners of such a rectangle thatoutlines the seated volume. Styling and other constraints often preventmounting of transducers at the optimum locations.

An optical infrared transmitter and receiver assembly is shown generallyat 52 in FIG. 8B and is mounted onto the instrument panel facing thewindshield. Assembly 52 can either be recessed below the upper face ofthe instrument panel or mounted onto the upper face of the instrumentpanel. Assembly 52, shown enlarged, comprises a source of infraredradiation, or another form of electromagnetic radiation, and a CCD, CMOSor other appropriate arrays of typically 160 pixels by 160 pixels. Inthis embodiment, the windshield is used to reflect the illuminationlight provided by the infrared radiation toward the objects in thepassenger compartment and also reflect the light being reflected back bythe objects in the passenger compartment, in a manner similar to the“heads-up” display which is now being offered on several automobilemodels. The “heads-up” display, of course, is currently used only todisplay information to the driver and is not used to reflect light fromthe driver to a receiver. Once again, unless one of the distancemeasuring systems as described below is used, this system alone cannotbe used to determine distances from the objects to the sensor. Its mainpurpose is object identification and monitoring. Depending on theapplication, separate systems can be used for the driver and for thepassenger. In some cases, the cameras located in the instrument panelwhich receive light reflected off of the windshield can be co-locatedwith multiple lenses whereby the respective lenses aimed at the driverand passenger seats respectively.

Assembly 52 is actually about two centimeters or less in diameter and isshown greatly enlarged in FIG. 8B. Also, the reflection area on thewindshield is considerably smaller than illustrated and specialprovisions are made to assure that this area of the windshield is flatand reflective as is done generally when heads-up displays are used. Forcases where there is some curvature in the windshield, it can be atleast partially compensated for by the CCD optics.

Transducers 23-25 are illustrated mounted onto the A-pillar of thevehicle, however, since these transducers are quite small, typicallyless than 2 cm on a side, they could alternately be mounted onto thewindshield itself, or other convenient location which provides a clearview of the portion of the passenger compartment being monitored. Otherpreferred mounting locations include the headliner above and also theside of the seat. Some imagers are now being made that are less than 1cm on a side.

FIG. 21 is a side view, with certain portions removed or cut away, of aportion of the passenger compartment of a vehicle showing preferredmounting locations of optical interior vehicle monitoring sensors(transmitter/receiver assemblies or transducers) 49, 50, 51, 54, 126,127, 128, 129, and 130. Each of these sensors is illustrated as having alens and is shown enlarged in size for clarity. In a typical actualdevice, the diameter of the lens is less than 2 cm and it protrudes fromthe mounting surface by less than 1 cm. Specially designed sensors canbe considerably smaller. This small size renders these devices almostunnoticeable by vehicle occupants. Since these sensors are optical, itis important that the lens surface remains relatively clean. Controlcircuitry 132, which is coupled to each transducer, contains aself-diagnostic feature where the image returned by a transducer iscompared with a stored image and the existence of certain key featuresis verified. If a receiver fails this test, a warning is displayed tothe driver which indicates that cleaning of the lens surface isrequired.

The technology illustrated in FIG. 21 can be used for numerous purposesrelating to monitoring of the space in the passenger compartment behindthe driver including: (i) the determination of the presence and positionof objects in the rear seat(s), (ii) the determination of the presence,position and orientation of child seats 2 in the rear seat, (iii) themonitoring of the rear of an occupant's head 33, (iv) the monitoring ofthe position of occupant 30, (v) the monitoring of the position of theoccupant's knees 35, (vi) the monitoring of the occupant's positionrelative to the airbag 44, (vii) the measurement of the occupant'sheight, as well as other monitoring functions as described herein.

Information relating to the space behind the driver can be obtained byprocessing the data obtained by the sensors 126, 127, 128 and 129, whichdata would be in the form of images if optical sensors are used as inthe preferred embodiment. Such information can be the presence of aparticular occupying item or occupant, e.g., a rear facing child seat 2as shown in FIG. 21, as well as the location or position of occupyingitems. Additional information obtained by the optical sensors caninclude an identification of the occupying item. The informationobtained by the control circuitry by processing the information fromsensors 126, 127, 128 and 129 may be used to affect any other system orcomponent in the vehicle in a similar manner as the information from thesensors which monitor the front seat is used as described herein, suchas the airbag system. Processing of the images obtained by the sensorsto determine the presence, position and/or identification of anyoccupants or occupying item can be effected using a pattern recognitionalgorithm in any of the ways discussed herein, e.g., a trained neuralnetwork. For example, such processing can result in affecting acomponent or system in the front seat such as a display that allows theoperator to monitor what is happening in the rear seat without having toturn his or her head.

In the preferred implementation, as shown in FIGS. 8A-8E, fourtransducer assemblies are positioned around the seat to be monitored,each can comprise one or more LEDs with a diverging lenses and a CMOSarray. Although illustrated together, the illuminating source in manycases will not be co-located with the receiving array. The LED emits acontrolled angle, 120° for example, diverging cone of infrared radiationthat illuminates the occupant from both sides and from the front andrear. This angle is not to be confused with the field angle used inultrasonic systems. With ultrasound, extreme care is required to controlthe field of the ultrasonic waves so that they will not create multipatheffects and add noise to the system. With infrared, there is no reason,in the implementation now being described, other than to make the mostefficient use of the infrared energy, why the entire vehicle cannot beflooded with infrared energy either from many small sources or from afew bright ones.

The image from each array is used to capture two dimensions of occupantposition information, thus, the array of assembly 50 positioned on thewindshield header, which is approximately 25% of the way laterallyacross the headliner in front of the driver, provides a both verticaland transverse information on the location of the driver. A similar viewfrom the rear is obtained from the array of assembly 54 positionedbehind the driver on the roof of the vehicle and above the seatbackportion of the seat 72. As such, assembly 54 also provides both verticaland transverse information on the location of the driver. Finally,arrays of assemblies 49 and 51 provide both vertical and longitudinaldriver location information. Another preferred location is the headlinercentered directly above the seat of interest. The position of theassemblies 49-52 and 54 may differ from that shown in the drawings. Inthe invention, in order that the information from two or more of theassemblies 49-52 and 54 may provide a three-dimensional image of theoccupant, or portion of the passenger compartment, the assembliesgenerally should not be arranged side-by-side. A side-by-sidearrangement as used in several prior art references discussed above,will provide two essentially identical views with the difference being alateral shift. This does not enable a complete three-dimensional view ofthe occupant.

One important point concerns the location and number of opticalassemblies. It is possible to use fewer than four such assemblies with apossible resulting loss in accuracy. The number of four was chosen sothat either a forward or rear assembly or either of the side assembliescan be blocked by a newspaper, for example, without seriously degradingthe performance of the system. Since drivers rarely are readingnewspapers while driving, fewer than four arrays are usually adequatefor the driver side. In fact, one is frequently sufficient. One camerais also usually sufficient for the passenger side if the goal of thesystem is classification only or if camera blockage is tolerated foroccupant tracking.

The particular locations of the optical assemblies were chosen to givethe most accurate information as to the locations of the occupant. Thisis based on an understanding of what information can be best obtainedfrom a visual image. There is a natural tendency on the part of humansto try to gauge distance from the optical sensors directly. This, as canbe seen above, is at best complicated involving focusing systems,stereographic systems, multiple arrays and triangulation, time of flightmeasurement, etc. What is not intuitive to humans is to not try toobtain this distance directly from apparatus or techniques associatedwith the mounting location. Whereas ultrasound is quite good formeasuring distances from the transducer (the z-axis), optical systemsare better at measuring distances in the vertical and lateral directions(the x and y-axes). Since the precise locations of the opticaltransducers are known, that is, the geometry of the transducer locationsis known relative to the vehicle, there is no need to try to determinethe displacement of an object of interest from the transducer (thez-axis) directly. This can more easily be done indirectly by anothertransducer. That is, the vehicle z-axis to one transducer is the camerax-axis to another.

The applications described herein have been illustrated using the driverof the vehicle. The same systems of determining the position of theoccupant relative to the airbag apply to the passenger, sometimesrequiring minor modifications. Also of course, a similar system can beappropriately designed for other monitoring situations such as for cargocontainers and truck trailers.

It is likely that the sensor required triggering time based on theposition of the occupant will be different for the driver than for thepassenger. Current systems are based primarily on the driver with theresult that the probability of injury to the passenger is necessarilyincreased either by deploying the airbag too late or by failing todeploy the airbag when the position of the driver would not warrant itbut the passenger's position would. With the use of occupant positionsensors for both the passenger and driver, the airbag system can beindividually optimized for each occupant and result in furthersignificant injury reduction. In particular, either the driver orpassenger system can be disabled if either the driver or passenger isout of position.

There is almost always a driver present in vehicles that are involved inaccidents where an airbag is needed. Only about 30% of these vehicles,however, have a passenger. If the passenger is not present, there isusually no need to deploy the passenger side airbag. The occupantposition sensor, when used for the passenger side with proper patternrecognition circuitry, can also ascertain whether or not the seat isoccupied, and if not, can disable the deployment of the passenger sideairbag and thereby save the cost of its replacement. A sophisticatedpattern recognition system could even distinguish between an occupantand a bag of groceries or a box, for example, which in some cargocontainer or truck trailer monitoring situations is desired. Finally,there has been much written about the out of position child who isstanding or otherwise positioned adjacent to the airbag, perhaps due topre-crash braking. The occupant position sensor described herein canprevent the deployment of the airbag in this situation.

3.1 Single Camera, Dual Camera with Single Light Source

Many automobile companies are opting to satisfy the requirements ofFMVSS-208 by using a weight only system such as the bladder or straingage systems disclosed here. Such a system provides an elementarymeasure of the weight of the occupying object but does not give areliable indication of its position, at least for automotive vehicles.It can also be easily confused by any object that weighs 60 or morepounds and that is interpreted as an adult. Weight only systems are alsostatic systems in that due to vehicle dynamics that frequently accompanya pre crash braking event they are unable to track the position of theoccupant. The load from seatbelts can confuse the system and therefore aspecial additional sensor must be used to measure seatbelt tension. Insome systems, the device must be calibrated for each vehicle and thereis some concern as to whether this calibration will be proper for thelife on the vehicle.

A single camera can frequently provide considerably more informationthan a weight only system without the disadvantages of weight sensorsand do so at a similar cost. Such a single camera in its simplestinstallation can categorize the occupancy state of the vehicle anddetermine whether the airbag should be suppressed due to an empty seator the presence of a child of a size that corresponds to one weighingless than 60 pounds. Of course, a single camera can also easily doconsiderably more by providing a static out-of-position indication and,with the incorporation of a faster processor, dynamic out-of-positiondetermination can also be provided. Thus, especially with the costs ofmicroprocessors continuing to drop, a single camera system can easilyprovide considerably more functionality than a weight only system andyet stay in the same price range.

A principal drawback of a single camera system is that it can be blockedby the hand of an occupant or by a newspaper, for example. This is arare event since the preferred mounting location for the camera istypically high in the vehicle such as on the headliner. Also, it isconsiderably less likely that the occupant will always be reading anewspaper, for example, and if he or she is not reading it when thesystem is first started up, or at any other time during the trip, thecamera system will still get an opportunity to see the occupant when heor she is not being blocked and make the proper categorization. Theability of the system to track the occupant will be impaired but thesystem can assume that the occupant has not moved toward the airbagwhile reading the newspaper and thus the initial position of theoccupant can be retained and used for suppression determination.Finally, the fact that the camera is blocked can be determined and thedriver made aware of this fact in much the same manner that a seatbeltlight notifies the driver that the passenger is not wearing his or herseatbelt.

The accuracy of a single camera system can be above 99% whichsignificantly exceeds the accuracy of weight only systems. Nevertheless,some automobile manufacturers desire even greater accuracy and thereforeopt for the addition of a second camera. Such a camera is usually placedon the opposite side of the occupant as the first camera. The firstcamera may be placed on or near the dome light, for example, and thesecond camera can be on the headliner above the side door. A dual camerasystem such as this can operate more accurately in bright daylightsituations where the window area needs to be ignored in the view of thecamera that is mounted near the dome.

Sometimes, in a dual camera system, only a single light source is used.This provides a known shadow pattern for the second camera and helps toaccentuate the edges of the occupying item rendering classificationeasier. Any of the forms of structured light can also be used andthrough these and other techniques the corresponding points in the twoimages can more easily be determined thus providing a three-dimensionalmodel of the occupant or occupying object in the case of other vehicletypes such as a cargo container or truck trailer.

As a result, the current assignee has developed a low cost single camerasystem which has been extensively tested for the most difficult problemof automobile occupant sensing but is nevertheless also applicable formonitoring of other vehicles such as cargo containers and trucktrailers. The automotive occupant position sensor system uses a CMOScamera in conjunction with pattern recognition algorithms for thediscrimination of out-of-position occupants and rear facing child safetyseats. A single imager, located strategically within the occupantcompartment, is coupled with an infrared LED that emits unfocused,wide-beam pulses toward the passenger volume. These pulses, whichreflect off of objects in the passenger seat and are captured by thecamera, contain information for classification and locationdetermination in approximately 10 msec. The decision algorithm processesthe returned information using a uniquely trained neural network, whichmay not be necessary in the simpler cargo container or truck trailermonitoring cases. The logic of the neural network was developed throughextensive in-vehicle training with thousands of realistic occupant sizeand position scenarios. Although the optical occupant position sensorcan be used in conjunction with other technologies (such as weightsensing, seat belt sensing, crash severity sensing, etc.), it is astand-alone system meeting the requirements of FMVSS-208. This devicewill be discussed in detail below.

3.2 Location of the Transducers

Any of the transducers discussed herein such as an active pixel or othercamera can be arranged in various locations in the vehicle including ina headliner, roof, ceiling, rear view mirror assembly, an A-pillar, aB-pillar and a C-pillar or a side wall or even a door in the case of acargo container or truck trailer. Images of the front seat area or therear seat area can be obtained by proper placement and orientation ofthe transducers such as cameras. The rear view mirror assembly can be agood location for a camera, particularly if it is attached to theportion of the mirror support that does not move when the occupant isadjusting the mirror. Cameras at this location can get a good view ofthe driver, passenger as well as the environment surrounding the vehicleand particularly in the front of the vehicle. It is an ideal locationfor automatic dimming headlight cameras.

3.3 Color Cameras—Multispectral Imaging

Most if not all occupant sensing systems, except those of the currentassignee, developed to date as reported in the patent and non-patentliterature have been generally based on a single frequency. As discussedherein, use of multiple frequencies with ultrasound makes it possible tochange a static system into a dynamic system allowing the occupant to betracked during pre-crash braking, for example. Multispectral imaging canalso provide advantages for camera or other optical-based systems. Thecolor of the skin of an occupant is a reliable measure of the presenceof an occupant and also renders the segmentation of the image to be moreeasily accomplished. Thus, the face can be more easily separated fromthe rest of the image simplifying the determination of the location ofthe eyes of the driver, for example. This is particularly true forvarious frequencies of passive and active infrared. Also, as discussedin more detail below, life forms react to radiation of differentfrequencies differently than non-life forms again making thedetermination of the presence of a life form easier. Finally, there isjust considerably more information in a color or multispectral imagethan in a monochromic image. This additional information improves theaccuracy of the identification and tracking process and thus of thesystem. In many cases, this accuracy improvement is so small that theadded cost is not justified but as costs of electronics and camerascontinue to drop this equation is changing and it is expected thatmultispectral imaging will prevail.

Illumination for nighttime is frequently done using infrared. Whenmultispectral imaging is used the designer has the choice of revertingto IR only for night time or using a multispectral LED and a verysensitive camera so that the flickering light does not annoy the driver.Alternately, a sensitive camera along with a continuous low level ofillumination can be used. Of course, multispectral imaging does notrequire that the visible part of the spectrum be used. Ultraviolet,X-rays and many other frequencies in the infrared part of the spectrumare available. Life forms, particularly humans, exhibit particularlyinteresting and identifiable reactions (reflection, absorption,scattering, transmission, emission) to frequencies in other parts of theelectromagnetic spectrum (see for example the book Alien Visionreferenced above) as discussed herein.

3.4 High Dynamic Range Cameras

An active pixel camera is a special camera which has the ability toadjust the sensitivity of each pixel of the camera similar to the mannerin which an iris adjusts the sensitivity of all of the pixels togetherof a camera. Thus, the active pixel camera automatically adjusts to theincident light on a pixel-by-pixel basis. An active pixel camera differsfrom an active infrared sensor in that an active infrared sensor, suchas of the type envisioned by Mattes et al. (discussed above), isgenerally a single pixel sensor that measures the reflection of infraredlight from an object. In some cases, as in the HDRC camera, the outputof each pixel is a logarithm of the incident light thus giving a highdynamic range to the camera. This is similar to the technique used tosuppress the effects of thermal gradient distortion of ultrasonicsignals as described in above-referenced patents. Thus, if the incidentradiation changes in magnitude by 1,000,000, for example, the output ofthe pixel may change by a factor of only 6.

A dynamic pixel camera is a camera having a plurality of pixels andwhich provides the ability to pick and choose which pixels should beobserved, as long as they are contiguous.

An HDRC camera is a type of active pixel camera where the dynamic rangeof each pixel is considerably broader. An active pixel cameramanufactured by the Photobit Corporation has a dynamic range of 70 dbwhile an IMS Chips camera, an HDRC camera manufactured by anothermanufacturer, has a dynamic range of 120 db. Thus, the HDRC camera has a100,000 times greater range of light sensitivity than the Photobitcamera.

The accuracy of the optical occupant sensor is dependent upon theaccuracy of the camera. The dynamic range of light within a vehicle canexceed 120 decibels. When a car is driving at night, for example, verylittle light is available whereas when driving in a bright sunlight,especially in a convertible, the light intensity can overwhelm manycameras. Additionally, the camera must be able to adjust rapidly tochanges in light caused by, for example, the emergence of the vehiclefrom tunnel, or passing by other obstructions such as trees, buildings,other vehicles, etc. which temporarily block the sun and can cause astrobing effect at frequencies approaching 1 kHz.

As mentioned, the IMS HDRC technology provides a 120 dB dynamicintensity response at each pixel in a monochromatic mode. The technologyhas a 1 million to one dynamic range at each pixel. This preventsblooming, saturation and flaring normally associated with CMOS and CCDcamera technology. This solves a problem that will be encountered in anautomobile when going from a dark tunnel into bright sunlight. Such arange can even exceed the 120 dB intensity.

There is also significant infrared radiation from bright sunlight andfrom incandescent lights within the vehicle. Such situations may evenexceed the dynamic range of the HDRC camera and additional filtering maybe required. Changing the bias on the receiver array, the use of amechanical iris, or of electrochromic glass or liquid crystal, or a Kerror Pockel cell can provide this filtering on a global basis but not at apixel level. Filtering can also be used with CCD arrays, but the amountof filtering required is substantially greater than for the HDRC camera.A notch filter can be used to block significant radiation from the sun,for example. This notch filter can be made as a part of the lens throughthe placement of various coatings onto the lens surface.

Liquid crystals operate rapidly and give as much as a dynamic range of10,000 to 1 but may create a pixel interference affect. Electrochromicglass operates more slowly but more uniformly thereby eliminating thepixel affect. The pixel effect arises whenever there is one pixel devicein front of another. This results in various aliasing, Moiré patternsand other ambiguities. One way of avoiding this is to blur the image.Another solution is to use a large number of pixels and combine groupsof pixels to form one pixel of information and thereby to blur the edgesto eliminate some of the problems with aliasing and Moiré patterns. Analternate to the liquid crystal device is the suspended particle deviceor SPD as discussed herein. Other alternatives include spatial lightmonitors such as Pockel or Kerr cells also discussed herein.

One straightforward approach is the use of a mechanical iris. Standardcameras already have response times of several tens of millisecondsrange. They will switch, for example, in a few frames on a typical videocamera (1 frame=0.033 seconds). This is sufficiently fast forcategorization but much too slow for dynamic out-of-position tracking.

An important feature of the IMS Chips HDRC camera is that the fulldynamic range is available at each pixel. Thus, if there are significantvariations in the intensity of light within the vehicle, and therebyfrom pixel to pixel, such as would happen when sunlight streams andthrough a window, the camera can automatically adjust and provide theoptimum exposure on a pixel by pixel basis. The use of the camera havingthis characteristic is beneficial to the invention described herein andcontributes significantly to system accuracy. CCDs have a rather limiteddynamic range due to their inherent linear response and consequentlycannot come close to matching the performance of human eyes. A keyadvantage of the IMS Chips HDRC camera is its logarithmic response whichcomes closest to matching that of the human eye. The IMS HDRC camera isalso useful in monitoring cargo containers and truck trailers where verylittle light is available when the door is shut. A small IR LED then canprovide the necessary light at a low power consumption which isconsistent with a system that may have to operate for long periods onbattery power.

Another approach, which is applicable in some vehicles at some times, isto record an image without the infrared illumination and then a secondimage with the infrared illumination and to then subtract the firstimage from the second image. In this manner, illumination caused bynatural sources such as sunlight or even from light bulbs within thevehicle can be subtracted out. Using the logarithmic pixel system of theIMS Chips camera, care must be taken to include the logarithmic effectduring the subtraction process. For some cases, natural illuminationsuch as from the sun, light bulbs within the vehicle, or radiationemitted by the object itself can be used alone without the addition of aspecial source of infrared illumination as discussed below.

Other imaging systems such as CCD arrays can also of course be used withat least one of the inventions disclosed herein. However, the techniqueswill be different since the camera is very likely to saturate whenbright light is present and to require the full resolution capability,when the light is dim, of the camera iris and shutter speed settings toprovide some compensation. Generally, when practicing at least one ofthe inventions disclosed herein, the interior of the passengercompartment will be illuminated with infrared radiation.

One novel solution is to form the image in memory by adding up asequence of very short exposures. The number stored in memory would bethe sum of the exposures on a pixel by pixel basis and the problem ofsaturation disappears since the memory location can be made as floatingpoint numbers. This then permits the maximum dynamic range but requiresthat the information from all of the pixels be removed at high speed. Insome cases, each pixel would then be zeroed while in others, the chargecan be left on the pixel since when saturation occurs the relevantinformation will already have been obtained.

There are other bright sources of infrared that must be accounted for.These include the sun and any light bulbs that may be present inside thevehicle. This lack of a high dynamic range inherent with the CCDtechnology requires the use of an iris, fast electronic shutter, liquidcrystal, Kerr or Pockel cell, or electrochromic glass filter to beplaced between the camera and the scene. Even with these filtershowever, some saturation can take place with CCD cameras under brightsun or incandescent lamp exposure. This saturation reduces the accuracyof the image and therefore the accuracy of the system. In particular,the training regimen that must be practiced with CCD cameras is moresevere since all of the saturation cases must be considered since thecamera may be unable to appropriately adjust. Thus, although CCD camerascan be used, HDRC logarithmic cameras such as manufactured by IMS Chipsare preferred. They not only provide a significantly more accurate imagebut also significantly reduce the amount of training effort andassociated data collection that must be undertaken during thedevelopment of the neural network algorithm or other computationalintelligence system. In some applications, it is possible to use othermore deterministic image processing or pattern recognition systems thanneural networks.

Another very important feature of the HDRC camera from IMS Chips is thatthe shutter time is constant at less than 100 ns irrespective ofbrightness of the scene. The pixel data arrives at constant ratesynchronous with the internal imager clock. Random access to each pixelfacilitates high-speed intelligent access to any sub-frame (block) sizeor sub-sampling ratio and a trade-off of frame speed and frame sizetherefore results. For example, a scene with 128 K pixels per frame canbe taken at 120 frames per second, or about 8 milliseconds per frame,whereas a sub-frame can be taken in run at as high as 4000 frames persecond with 4 K pixels per frame. This combination allows the maximumresolution for the identification and classification part of theoccupant sensor problem while permitting a concentration on thoseparticular pixels which track the head or chest, as described above, fordynamic out-of-position tracking. In fact, the random access features ofthese cameras can be used to track multiple parts of the imagesimultaneously while ignoring the majority of the image, and do so atvery high speed. For example, the head can be tracked simultaneouslywith the chest by defining two separate sub-frames that need not beconnected. This random access pixel capability, therefore, is optimallysuited for recognizing and tracking vehicle occupants. It is also suitedfor monitoring the environment outside of the vehicle for the purposesof blind spot detection, collision avoidance and anticipatory sensing.Photobit Corporation of 135 North Los Robles Ave., Suite 700, Pasadena,Calif. 91101 manufactures a camera with some characteristics similar tothe IMS Chips camera. Other competitive cameras can be expected toappear on the market.

Photobit refers to their Active Pixel Technology as APS. According toPhotobit, in the APS, both the photo detector and readout amplifier arepart of each pixel. This allows the integrated charge to be convertedinto a voltage in the pixel that can then be read out over X-Y wiresinstead of using a charge domain shift register as in CCDs. This columnand row addressability (similar to common DRAM) allows for window ofinterest readout (windowing) which can be utilized for on chipelectronic pan/tilt and zoom. Windowing provides added flexibility inapplications, such as disclosed herein, needing image compression,motion detection or target tracking. The APS utilizes intra-pixelamplification in conjunction with both temporal and fixed pattern noisesuppression circuitry (i.e., correlated double sampling), which producesexceptional imagery in terms of wide dynamic range (˜75 dB) and lownoise (˜15 e-rms noise floor) with low fixed pattern noise (<0.15% sat).Unlike CCDs, the APS is not prone to column streaking due to bloomingpixels. This is because CCDs rely on charge domain shift registers thatcan leak charge to adjacent pixels when the CCD registers overflows.Thus, bright lights “bloom” and cause unwanted streaks in the image. Theactive pixel can drive column busses at much greater rates than passivepixel sensors and CCDs. On-chip analog-to-digital conversion (ADC)facilitates driving high speed signals off chip. In addition, digitaloutput is less sensitive to pickup and crosstalk, facilitating computerand digital controller interfacing while increasing system robustness. Ahigh speed APS recently developed for a custom binary output applicationproduced over 8,000 frames per second, at a resolution of 128×128pixels. It is possible to extend this design to a 1024×1024 array sizeand achieve greater than 1000 frames per second for machine vision. Allof these features can be important to many applications of at least oneof the inventions disclosed herein.

These advanced cameras, as represented by the HDRC and the APS cameras,now make it possible to more accurately monitor the environment in thevicinity of the vehicle. Previously, the large dynamic range ofenvironmental light has either blinded the cameras when exposed tobright light or else made them unable to record images when the lightlevel was low. Even the HDRC camera with its 120 dB dynamic range may bemarginally sufficient to handle the fluctuations in environmental lightthat occur. Thus, the addition of a electrochromic, liquid crystal, SPD,spatial light monitors or other similar filter may be necessary. This isparticularly true for cameras such as the Photobit APS camera with its75 dB dynamic range.

At about 120 frames per second, these cameras are adequate for caseswhere the relative velocity between vehicles is low. There are manycases, however, where this is not the case and a much higher monitoringrate is required. This occurs for example, in collision avoidance andanticipatory sensor applications. The HDRC camera is optimally suitedfor handling these cases since the number of pixels that are beingmonitored can be controlled resulting in a frame rate as high as about4000 frames per second with a smaller number of pixels.

Another key advantage of the HDRC camera is that it is quite sensitiveto infrared radiation in the 0.8 to 1 micron wavelength range. Thisrange is generally beyond visual range for humans permitting this camerato be used with illumination sources that are not visible to the humaneye. A notch filter is frequently used with the camera to eliminateunwanted wavelengths. These cameras are available from the Institute forMicroelectronics (IMS Chips), Allamndring 30a, D-70569 Stuttgart,Germany with a variety of resolutions ranging from 512 by 256 to 720 by576 pixels and can be custom fabricated for the resolution and responsetime required.

One problem with high dynamic range cameras, particularly those makinguse of a logarithmic compression is that the edges of objects in thefield of view tend to wash out and the picture loses a lot of contrast.This causes problems for edge detecting algorithms and thus reduces theaccuracy of the system. There are a number of other different methods ofachieving a high dynamic range without sacrificing contrast. One systemby Nayar, as discussed herein, takes a picture using adjacent pixelswith different radiation blocking filers. Four such pixel types are usedallowing Nayar to essentially obtain 4 separate pictures with one snapof the shutter. Software then selects which of the four pixels to usefor each part of the image so that the dark areas receive one exposureand somewhat brighter areas another exposure and so on. The brightestpixel receives all of the incident light, the next brightest filtershalf of the light, the next brightest half again and the dullest pixelhalf again. Other ratios could be used as could more levels of pixels,e.g., eight instead of four. Experiments have shown that this issufficient to permit a good picture to be taken when bright sunlight isstreaming into a dark room. A key advantage of this system is that thefull frame rate is available and the disadvantage is that only 25% ofthe pixels are in fact used to form the image.

Another system drains the charge off of the pixels as the picture isbeing taken and stored the integrated results in memory. TFA technologylends itself to this implementation. As long as the memory capacity issufficient, the pixel never saturates. An additional approach is to takemultiple images at different iris or shutter settings and combine themin much the same way as with the Nayar method. A still differentapproach is to take several pictures at a short shutter time or a smalliris setting and combine the pictures in a processor or otherappropriate device. In this manner, the effective dynamic range of thecamera can be extended. This method may be too slow for some dynamicapplications.

3.5 Fisheye Lens, Pan and Zoom

Infrared waves are shown coming from the front and back transducerassemblies 54 and 55 in FIG. 8C. FIG. 8D illustrates two optical systemseach having a source of infrared radiation and a CCD, CMOS, FPR, TFA orQWIP array receiver. The price of such arrays has dropped dramaticallyrecently making most of them practical for interior and exterior vehiclemonitoring. In this embodiment, transducers 54 and 55 are CMOS arrayshaving 160 pixels by 160 pixels covered by a lens. In some applications,this can create a “fisheye” effect whereby light from a wide variety ofdirections can be captured. One such transducer placed by the dome lightor other central position in the vehicle headliner, such as thetransducer designated 54, can monitor the entire vehicle interior withsufficient resolution to determine the occupancy of the vehicle, forexample. Imagers such as those used herein are available from MarshallElectronics Inc. of Culver City, Calif. and others. A fisheye lens is “. . . a wide-angle photographic lens that covers an angle of about 180°,producing a circular image with exaggerated foreshortening in the centerand increasing distortion toward the periphery”. (The American HeritageDictionary of the English Language, Third Edition, 1992 by HoughtonMifflin Company). This distortion of a fisheye lens can be substantiallychanged by modifying the shape of the lens to permit particular portionsof the interior passenger compartment to be observed. Also, in manycases the full 180° is not desirable and a lens which captures a smallerangle may be used. Although primarily spherical lenses are illustratedherein, it is understood that the particular lens design will depend onthe location in the vehicle and the purpose of the particular receiver.A fisheye lens can be particularly useful for some truck trailer, cargocontainer, railroad car and automobile trunk monitoring cases.

A camera that provides for pan and zoom using a fisheye lens isdescribed in U.S. Pat. No. 5,185,667 and is applicable to at least oneof the inventions disclosed herein. Here, however, it is usually notnecessary to remove the distortion since the image will in general notbe viewed by a human but will be analyzed by software. One exception iswhen the image is sent to emergency services via telematics. In thatcase, the distortion removal is probably best done at the EMS site.

Although a fisheye camera has primarily been discussed above, othertypes of distorting lenses or mirrors can be used to accomplishedparticular objectives. A distorting lens or mirror, for example, canhave the effect of dividing the image into several sub-pictures so thatthe available pixels can cover more than one area of a vehicle interioror exterior. Alternately, the volume in close proximity to an airbag,for example, can be allocated a more dense array of pixels so thatmeasurements of the location of an occupant relative to the airbag canbe more accurately achieved. Numerous other objectives can now beenvisioned which can now be accomplished with a reduction in the numberof cameras or imagers through either distortion or segmenting of theoptical field.

Another problem associated with lens is cleanliness. In general, theoptical systems of these inventions comprise methods to test for thevisibility through the lens and issue a warning when that visibilitybegins to deteriorate. Many methods exist for accomplishing this featincluding the taking of an image when the vehicle is empty and notmoving and at night. Using neural networks, for example, or some othercomparison technique, a comparison of the illumination reaching theimager can be compared with what is normal. A network can be trained onempty seats, for example, in all possible positions and compared withthe new image. Or, those pixels that correspond to any movable surfacein the vehicle can be removed from the image and a brightness test onthe remaining pixels used to determine lens cleanliness.

Once a lens has been determined to be dirty, then either a warning lightcan be set telling the operator to visit the dealer or a method ofcleaning the lens automatically invoked. One such method for nightvision systems is disclosed in WO0234572. Another, which is one on theinventions disclosed herein, is to cover the lens with a thin film. Thisfilm may be ultrasonically excited thereby greatly minimizing thetendency for it to get dirty and/or the film can be part of a roll offilm that is advanced when the diagnostic system detects a dirty lensthereby placing a new clean surface in front of the imager. The filmroll can be sized such that under normal operation, the roll would lastsome period such as 20 years. A simple, powerless mechanism can bedesigned that will gradually advance the film across the lens over aperiod of 10 to 20 years using the normal daily thermal cycling to causerelative expansion and contraction of materials with differing thermalexpansion coefficients.

4. 3D Cameras

Optical sensors can be used to obtain a three-dimensional measurement ofthe object through a variety of methods that use time of flight,modulated light and phase measurement, quantity of light received withina gated window, structured light and triangulation etc. Some of thesetechniques are discussed in U.S. Pat. No. 6,393,133 and below.

4.1 Stereo

One method of obtaining a three-dimensional image is illustrated in FIG.8D wherein transducer 24 is an infrared source having a widetransmission angle such that the entire contents of the front driver'sseat is illuminated. Receiving imager transducers 23 and 25 are shownspaced apart so that a stereographic analysis can be made by the controlcircuitry 20. This circuitry 20 contains a microprocessor withappropriate pattern recognition algorithms along with other circuitry asdescribed above. In this case, the desired feature to be located isfirst selected from one of the two returned images from either imagingtransducer 23 or 25. The software then determines the location of thesame feature, through correlation analysis or other methods, on theother image and thereby, through analysis familiar to those skilled inthe art, determines the distance of the feature from the transducers bytriangulation.

As the distance between the two or more imagers used in the stereoconstruction increases, a better and better model of the object beingimaged can be obtained since more of the object is observable. On theother hand, it becomes increasingly difficult to pair up points thatoccur in both images. Given sufficient computational resources, this nota difficult problem but with limited resources and the requirement totrack a moving occupant during a crash, for example, the problem becomesmore difficult. One method to ease the problem is to project onto theoccupant, a structured light that permits a recognizable pattern to beobserved and matched up in both images. The source of this projectionshould lie midway between the two imagers. By this method, a rapidcorrespondence between the images can be obtained.

On the other hand, if a source of structured light is available at adifferent location than the imager, then a simpler three-dimensionalimage can be obtained using a single imager. Furthermore, the model ofthe occupant really only needs to be made once during the classificationphase of the process and there is usually sufficient time to accomplishthat model with ordinary computational power. Once the model has beenobtained, then only a few points need be tracked by either one or bothof the cameras.

Another method exists whereby the displacement between two images fromtwo cameras is estimated using a correlator. Such a fast correlator hasbeen developed by Professor Lukin of Kyiv, Ukraine in conjunction withhis work on noise radar. This correlator is very fast and can probablydetermine the distance to an occupant at a rate sufficient for trackingpurposes.

4.2 Distance by Focusing

In the above-described imaging systems, a lens within a receptorcaptures the reflected infrared light from the head or chest of thedriver, or other object to be monitored, and displays it onto an imagingdevice (CCD, CMOS, FPA, TFA, QWIP or equivalent) array. For thediscussion of FIGS. 5 and 13-17 at least, either CCD or the word“imager” will be used to include all devices which are capable ofconverting light frequencies, including infrared, into electricalsignals. In one method of obtaining depth from focus, the CCD is scannedand the focal point of the lens is altered, under control of anappropriate circuit, until the sharpest image of the driver's head orchest, or other object, results and the distance is then known from thefocusing circuitry. This trial and error approach may require the takingof several images and thus may be time consuming and perhaps too slowfor occupant tracking during pre-crash braking.

The time and precision of this measurement is enhanced if two receptors(e.g., lenses) are used which can either project images onto a singleCCD or onto separate CCDs. In the first case, one of the lenses could bemoved to bring the two images into coincidence while in the other case,the displacement of the images needed for coincidence would bedetermined mathematically. Other systems could be used to keep track ofthe different images such as the use of filters creating differentinfrared frequencies for the different receptors and again using thesame CCD array. In addition to greater precision in determining thelocation of the occupant, the separation of the two receptors can alsobe used to minimize the effects of hands, arms or other extremitieswhich might be very close to the airbag. In this case, where thereceptors are mounted high on the dashboard on either side of thesteering wheel, an arm, for example, would show up as a thin object butmuch closer to the airbag than the larger body parts and, therefore,easily distinguished and eliminated, permitting the sensors to determinethe distance to the occupant's chest. This is one example of the use ofpattern recognition.

An alternate method is to use a lens with a short focal length. In thiscase, the lens is mechanically focused, e.g., automatically, directly orindirectly, by the control circuitry 20, to determine the clearest imageand thereby obtain the distance to the object. This is similar tocertain camera auto-focusing systems such as one manufactured by Fuji ofJapan. Again this is a time consuming method. Other methods can be usedas described in patents and patent applications referenced above.

Instead of focusing the lens, the lens could be moved relative to thearray to thereby adjust the image on the array. Instead of moving thelens, the array could be moved to achieve the proper focus. In addition,it is also conceivable that software could be used to focus the imagewithout moving the lens or the array especially if at least two imagesare available.

An alternative is to use the focusing systems described in U.S. Pat.Nos. 5,193,124 and 5,003,166. These systems are quite efficientrequiring only two images with different camera settings. Thus, if thereis sufficient time to acquire an image, change the camera settings andacquire a second image, this system is fine and can be used with theinventions disclosed herein. Once the position of the occupant has beendetermined for one point in time, then the process may not have to berepeated as a measurement of the size of a part of an occupant can serveas a measure of its relative location compared to the previous imagefrom which the range was obtained. Thus, other than the requirement of asomewhat more expensive imager, the system of the '124 and '166 patentsis fine. The accuracy of the range is perhaps limited to a fewcentimeters depending on the quality of the imager used. Also, ifmultiple ranges to multiple objects are required, then the processbecomes a bit more complicated.

4.3 Ranging

The scanning portion of a pulse laser radar device can be accomplishedusing rotating mirrors, vibrating mirrors, or preferably, a solid statesystem, for example one utilizing TeO₂ as an optical diffraction crystalwith lithium niobate crystals driven by ultrasound (although other solidstate systems not necessarily using TeO₂ and lithium niobate crystalscould also be used) which is an example of an acoustic optical scanner.An alternate method is to use a micromachined mirror, which is supportedat its center and caused to deflect by miniature coils or equivalentMEMS device. Such a device has been used to provide two-dimensionalscanning to a laser. This has the advantage over the TeO₂-lithiumniobate technology in that it is inherently smaller and lower cost andprovides two-dimensional scanning capability in one small device. Themaximum angular deflection that can be achieved with this process is onthe order of about 10 degrees. Thus, a diverging lens or equivalent willbe needed for the scanning system.

Another technique to multiply the scanning angle is to use multiplereflections off of angled mirror surfaces. A tubular structure can beconstructed to permit multiple interior reflections and thus amultiplying effect on the scan angle.

An alternate method of obtaining three-dimensional information from ascanning laser system is to use multiple arrays to replace the singlearrays used in FIG. 8A. In the case, the arrays are displaced from eachother and, through triangulation, the location of the reflection fromthe illumination by a laser beam of a point on the object can bedetermined in a manner that is understood by those skilled in the art.Alternately, a single array can be used with the scanner displaced fromthe array.

A new class of laser range finders has particular application here. Thisproduct, as manufactured by Power Spectra, Inc. of Sunnyvale, Calif., isa GaAs pulsed laser device which can measure up to 30 meters with anaccuracy of <2 cm and a resolution of <1 cm. This system can beimplemented in combination with transducer 24 and one of the receivingtransducers 23 or 25 may thereby be eliminated. Once a particularfeature of an occupying item of the passenger compartment has beenlocated, this device is used in conjunction with an appropriate aimingmechanism to direct the laser beam to that particular feature. Thedistance to that feature can then be known to within 2 cm and withcalibration even more accurately. In addition to measurements within thepassenger compartment, this device has particular applicability inanticipatory sensing and blind spot monitoring applications exterior tothe vehicle. An alternate technology using range gating to measure thetime of flight of electromagnetic pulses with even better resolution canbe developed based on the teaching of the McEwan patents listed above.

A particular implementation of an occupant position sensor having arange of from 0 to 2 meters (corresponding to an occupant position offrom 0 to 1 meter since the signal must travel both to and from theoccupant) using infrared is illustrated in the block diagram schematicof FIG. 14. This system was designed for automobile occupant sensing anda similar system having any reasonable range up to and exceeding 100meters can be designed on the same principles for other monitoringapplications. The operation is as follows. A 48 MHz signal, f1, isgenerated by a crystal oscillator 81 and fed into a frequency tripler 82which produces an output signal at 144 MHz. The 144 MHz signal is thenfed into an infrared diode driver 83 which drives the infrared diode 84causing it to emit infrared light modulated at 144 MHz and a referencephase angle of zero degrees. The infrared diode 84 is directed at thevehicle occupant. A second signal f2 having a frequency of 48.05 MHz,which is slightly greater than f1, is similarly fed from a crystaloscillator 85 into a frequency tripler 86 to create a frequency of144.15 MHz. This signal is then fed into a mixer 87 which combines itwith the 144 MHz signal from frequency tripler 82. The combined signalfrom the mixer 87 is then fed to filter 88 which removes all signalsexcept for the difference, or beat frequency, between 3 times f1 and 3times f2, of 150 kHz. The infrared signal which is reflected from theoccupant is received by receiver 89 and fed into pre-amplifier 91, aresistor 90 to bias being coupled to the connection between the receiver89 and the pre-amplifier 91. This signal has the same modulationfrequency, 144 MHz, as the transmitted signal but now is out of phasewith the transmitted signal by an angle x due to the path that thesignal took from the transmitter to the occupant and back to thereceiver.

The output from pre-amplifier 91 is fed to a second mixer 92 along withthe 144.15 MHz signal from the frequency tripler 86. The output frommixer 92 is then amplified by an automatic gain amplifier 93 and fedinto filter 94. The filter 94 eliminates all frequencies except for the150 kHz difference, or beat, frequency, in a similar manner as was doneby filter 88. The resulting 150 kHz frequency, however, now has a phaseangle x relative to the signal from filter 88. Both 150 kHz signals arenow fed into a phase detector 95 which determines the magnitude of thephase angle x. It can be shown mathematically that, with the abovevalues, the distance from the transmitting diode to the occupant isx/345.6 where x is measured in degrees and the distance in meters. Thevelocity can also be obtained using the distance measurement asrepresented by 96. An alternate method of obtaining distanceinformation, as discussed above, is to use the teachings of the McEwanpatents discussed herein.

As reported above, cameras can be used for obtaining three-dimensionalimages by modulation of the illumination as taught in U.S. Pat. No.5,162,861. Use of a ranging device for occupant sensing is believed tohave been first disclosed by the current assignee. More recent attemptsinclude the PMD camera as disclosed in PCT application WO09810255 andsimilar concepts disclosed in U.S. Pat. Nos. 6,057,909 and 6,100,517.

Note that although the embodiment in FIG. 14 uses near infrared, it ispossible to use other frequencies of energy without deviating from thescope of the invention. In particular, there are advantages in using theshort wave (SWIR), medium wave (MWIR) and long wave (LWIR) portions ofthe infrared spectrum as the interact in different and interesting wayswith living occupants as described herein and in the book Alien Visionreferenced above.

4.4 Pockel or Kerr Cell for Determining Range

Pockel and Kerr cells are well known in optical laboratories. They actas very fast shutters (up to 10 billion cycles per second) and as suchcan be used to range-gate the reflections based on distance giving arange resolution of up to 3 cm without the use of phase techniques todivide the interval into parts or sub millimeter resolution usingphasing techniques. Thus, through multiple exposures the range to allreflecting surfaces inside and outside of the vehicle can be determinedto any appropriate degree of accuracy. The illumination is transmitted,the camera shutter opened and the cell allows only that reflected lightto enter the camera that arrived at the cell a precise time range afterthe illumination was initiated.

These cells are part of a class of devices called spatial lightmodulators (SLM). One novel application of an SLM is reported in U.S.Pat. No. 5,162,861. In this case, an SLM is used to modulate the lightreturning from a transmitted laser pulse that is scattered from atarget. By comparing the intensities of the modulated and unmodulatedimages, the distance to the target can be ascertained. Using a SLM inanother manner, the light valve can be kept closed for all ranges exceptthe ones of interest. By changing the open time of the SLM, only returnsfrom certain distances are permitted to pass through to the imager. Byselective changing the opened time, the range to the target can be“range-gated” and thereby accurately determined. Thus, the outgoinglight need not be modulated and a scanner is not necessary unless thereis a need to overcome the power of the sun reflecting off of the objectof interest. This form of range-gating can of course be used for eitherexternal or internal applications.

4.5 Thin film on ASIC (TFA)

Since the concepts of using cameras for monitoring the passengercompartment of a vehicle and measuring distance to a vehicle occupantbased on the time of flight were first disclosed in commonly assignedabove-referenced patents, several improvements have been reported in theliterature including the thin film on ASIC (TFA) (references 6-11) andphotonic mixing device (PMD) (reference 12) camera technologies. Both ofthese technologies and combinations thereof are good examples of devicesthat can be used in practicing the inventions herein and those inabove-referenced patents and applications for monitoring both inside andexterior to a vehicle.

An improvement to these technologies is to use noise or pseudo noisemodulation for a PMD-like device to permit more accurate distance toobject determination especially for exterior to the vehicle monitoringthrough correlation of the generated and reflected modulation sequences.This has the further advantage that systems from different vehicles willnot interfere with each other.

The TFA is an example of a high dynamic range camera (HDRC) the use ofwhich for interior monitoring was disclosed in U.S. Pat. No. 6,393,133.Since there is direct connection between each pixel and an associatedelectronic circuit, the potential exists for range gating the sensor toisolate objects between certain limits thus simplifying theidentification process by eliminating reflections from objects that arecloser or further away than the object of interest. A further advantageof the TFA is that it can be doped to improve its sensitivity toinfrared and it also can be fabricated as a three-color camera system.

Another novel HDRC camera is disclosed by Nayar (reference 13), asdiscussed above, and involves varying the sensitivity of pixels in theimager. Each of four adjacent pixels has a different exposuresensitivity and an algorithm is presented that combines the fourexposures in a manner that loses little resolution but provides a highdynamic range picture. This particularly simple system is a preferredapproach to handling the dynamic range problem in several monitoringapplications of at least one of the inventions disclosed herein.

A great deal of development effort has gone into automatic camerafocusing systems such as described in the Scientific American Article“Working Knowledge: Focusing in a Flash” (reference 14). The technologyis now to the point that it can be taught to focus on a particularobject, such as the head or chest of an occupant, or other object, andmeasure the distance to the object to within approximately 1 inch. Ifthis technology is coupled with the Nayar camera, a very low cost semi3D high dynamic range camera or imager results that is sufficientlyaccurate for locating an occupant in the passenger compartment or anobject in another container. If this technology is coupled with an eyelocator and the distance to the eyes of the occupant are determined,then a single camera is all that is required for either the driver orpassenger. Such a system would display a fault warning when it is unableto find the occupant's eyes.

As discussed above, thin film on ASIC technology, as described in Lake,D. W. “TFA Technology: The Coming Revolution in Photography”, AdvancedImaging Magazine, April, 2002 (www.advancedimagingmag.com) shows promiseof being the next generation of imager for automotive and other vehiclemonitoring applications. The anticipated specifications for thistechnology, as reported in the Lake article, are:

Dynamic Range 120 db Sensitivity 0.01 lux Anti-blooming 1,000,000:1Pixel Density 3,200,000 Pixel Size 3.5 um Frame Rate 30 fps DC Voltage1.8 v Compression 500 to 1

All of these specifications, except for the frame rate, are attractivefor occupant sensing. It is believed that the frame rate can be improvedwith subsequent generations of the technology. Some advantages of thistechnology for occupant sensing include the possibility of obtaining athree-dimensional image by varying the pixel on time in relation to amodulated illumination in a simpler manner than that proposed with thePMD imager or with a Pockel or Kerr cell. The ability to build theentire package on one chip will reduce the cost of this imager comparedwith two or more chips required by current technology. Other technicalpapers on TFA are referenced above.

TFA thus appears to be a major breakthrough when used in the interiorand exterior imaging systems. Its use in these applications falls withinthe teachings of the inventions disclosed herein.

5. Glare Control

The headlights of oncoming vehicles frequently make it difficult for thedriver of a vehicle to see the road and safely operate the vehicle. Thisis a significant cause of accidents and much discomfort. The problem isespecially severe during bad weather where rain can cause multiplereflections. Opaque visors are now used to partially solve this problembut they do so by completely blocking the view through a large portionof the window and therefore cannot be used to cover the entirewindshield. Similar problems happen when the sun is setting or risingand the driver is operating the vehicle in the direction of the sun.U.S. Pat. No. 4,874,938 attempts to solve this problem through the useof a motorized visor but although it can block some glare sources, italso blocks a substantial portion of the field of view.

The vehicle interior monitoring system disclosed herein can contributeto the solution of this problem by determining the position of thedriver's eyes. If separate sensors are used to sense the direction ofthe light from the on-coming vehicle or the sun, and through the use ofelectrochromic glass, a liquid crystal device, suspended particle deviceglass (SPD) or other appropriate technology, a portion of thewindshield, or special visor, can be darkened to impose a filter betweenthe eyes of the driver and the light source. Electrochromic glass is amaterial where the transparency of the glass can be changed through theapplication of an electric current. The term “liquid crystal” as usedherein will be used to represent the class of all such materials wherethe optical transmissibility can be varied electrically orelectronically. Electrochromic products are available from Gentex ofZeeland, Mich., and Donnelly of Holland, Mich. Other systems forselectively imposing a filter between the eyes of an occupant and thelight source are currently under development.

By dividing the windshield into a controlled grid or matrix ofcontiguous areas and through feeding the current into the windshieldfrom orthogonal directions, selective portions of the windshield can bedarkened as desired. Other systems for selectively imposing a filterbetween the eyes of an occupant and the light source are currently underdevelopment. One example is to place a transparent sun visor type devicebetween the windshield and the driver to selectively darken portions ofthe visor as described above for the windshield.

5.1 Windshield

FIG. 22 illustrates how such a system operates for the windshield. Asensor 135 located on vehicle 136 determines the direction of the light138 from the headlights of oncoming vehicle 137. Sensor 135 is comprisedof a lens and a charge-coupled device (CCD), CMOS or similar device,with appropriate software or electronic circuitry that determines whichelements of the CCD are being most brightly illuminated. An algorithmstored in processor 20 then calculates the direction of the light fromthe oncoming headlights based on the information from the CCD, or CMOSdevice. Usually two systems 135 are required to fix the location of theoffending light. Transducers 6, 8 and 10 determine the probable locationof the eyes of the operator 30 of vehicle 136 in a manner such asdescribed above and below. In this case, however, the determination ofthe probable locus of the driver's eyes is made with an accuracy of adiameter for each eye of about 3 inches (7.5 cm). This calculationsometimes will be in error especially for ultrasonic occupant sensingsystems and provision is made for the driver to make an adjustment tocorrect for this error as described below.

The windshield 139 of vehicle 136 comprises electrochromic glass, aliquid crystal, SPD device or similar system, and is selectivelydarkened at area 140, FIG. 22A, due to the application of a currentalong perpendicular directions 141 and 142 of windshield 139. Theparticular portion of the windshield to be darkened is determined byprocessor 20. Once the direction of the light from the oncoming vehicleis known and the locations of the driver's eyes are known, it is amatter of simple trigonometry to determine which areas of the windshieldmatrix should be darkened to impose a filter between the headlights andthe driver's eyes. This is accomplished by the processor 20. A separatecontrol system, not shown, located on the instrument panel, steeringwheel or at some other convenient location, allows the driver to selectthe amount of darkening accomplished by the system from no darkening tomaximum darkening. In this manner, the driver can select the amount oflight that is filtered to suit his particular physiology. Alternately,this process can take place automatically. The sensor 135 can either bedesigned to respond to a single light source or to multiple lightsources to be sensed and thus multiple portions of the vehiclewindshield 139 to be darkened. Unless the camera is located on the sameaxis at the eyes of the driver, two cameras would in general be requiredto determine the distance of the glare causing object from the eyes ofthe driver. Without this third dimension, two glare sources that are onthe same axis to the camera could be on different axes to the driver,for example.

As an alternative to locating the direction of the offending lightsource, a camera looking at the eyes of the driver can determine whenthey are being subjected to glare and then impose a filter. A trial anderror process or through the use of structured light created by apattern on the windshield, determines where to create the filter toblock the glare.

More efficient systems are now becoming available to permit asubstantial cost reduction as well as higher speed selective darkeningof the windshield for glare control. These systems permit covering theentire windshield which is difficult to achieve with LCDs. For example,such systems are made from thin sheets of plastic film, sometimes withan entrapped liquid, and can usually be sandwiched between the twopieces of glass that make up a typical windshield. The development ofconductive plastics permits the addressing and thus the manipulation ofpixels of a transparent film that previously was not possible. These newtechnologies will now be discussed.

If the objective is for glare control, then the Xerox Gyricon technologyapplied to windows can be appropriate. Previously, this technology hasonly been used to make e-paper and a modification to the technology isnecessary for it to work for glare control. Gyricon is a thin layer oftransparent plastic full of millions of small black and white or red andwhite beads, like toner particles. The beads are contained in anoil-filled cavity. When voltage is applied, the beads rotate to presenta colored side to the viewer. The advantages of Gyricon are: (1) it iselectrically writeable and erasable; (2) it can be re-used thousands oftimes; (3) it does not require backlighting or refreshing; (4) it isbrighter than today's reflective displays; and, (5) it operates on lowpower. The changes required are to cause the colored spheres to rotate90 degrees rather than 180 degrees and to make half of each spheretransparent so that the display switches from opaque to 50% transparent.

Another technology, SPD light control technology from Research FrontiersInc., has been used to darken entire windows but not as a system fordarkening only a portion of the glass or sun visor to impose a selectivefilter to block the sun or headlights of an oncoming vehicle. Althoughit has been used as a display for laptop computers, it has not been usedas a heads-up display (HUD) replacement technology for automobile ortruck windshields.

Both SPD and Gyricon technologies require that the particles be immersedin a fluid so that the particles can move. Since the properties of thefluid will be temperature sensitive, these technologies will varysomewhat in performance over the automotive temperature range. Thepreferred technology, therefore, is plastic electronics although in manyapplications either Gyricon or SPD will also be used in combination withplastic electronics, at least until the technology matures. Currentlyplastic electronics can only emit light and not block it. However,research is ongoing to permit it to also control the transmission oflight.

The calculations of the location of the driver's eyes using acousticsystems may be in error and therefore provision must be made to correctfor this error. One such system permits the driver to adjust the centerof the darkened portion of the windshield to correct for such errorsthrough a knob, mouse pad, joy stick or other input device, on theinstrument panel, steering wheel, door, armrest or other convenientlocation. Another solution permits the driver to make the adjustment byslightly moving his head. Once a calculation as to the location of thedriver's eyes has been made, that calculation is not changed even thoughthe driver moves his head slightly. It is assumed that the driver willonly move his head in a very short time period to center the darkenedportion of the windshield to optimally filter the light from theoncoming vehicle. The monitoring system will detect this initial headmotion and make the correction automatically for future calculations.Additionally, a camera observing the driver or other occupant canmonitor the reflections of the sun or the headlights of oncomingvehicles off of the occupant's head or eyes and automatically adjust thefilter in the windshield or sun visor.

5.2 Glare in Rear View Mirrors

Electrochromic glass is currently used in rear view mirrors to darkenthe entire mirror in response to the amount of light striking anassociated sensor. This substantially reduces the ability of the driverto see objects coming from behind his vehicle. If one rear-approachingvehicle, for example, has failed to dim his lights, the mirror will bedarkened to respond to the light from that vehicle making it difficultfor the driver to see other vehicles that are also approaching from therear. If the rear view mirror is selectively darkened on only thoseportions that cover the lights from the offending vehicle, the driver isable to see all of the light coming from the rear whether the source isbright or dim. This permits the driver to see all of the approachingvehicles not just the one with bright lights.

Such a system is illustrated in FIGS. 23, 23A and 23B wherein rear viewmirror 55 is equipped with electrochromic glass, or comprises a liquidcrystal or similar device, having the capability of being selectivelydarkened, e.g., at area 143. Associated with mirror 55 is a light sensor144 that determines the direction of light 138 from the headlights ofrear approaching vehicle 137. Again, as with the windshield, a stereocamera is used if the camera is not aligned with the eye view path. Thisis easier to accomplish with a mirror due to its much smaller size. Insuch a case, the imager could be mounted on the movable part of themirror and could even look through the mirror from behind. In the samemanner as above, transducers 6, 8 and 10 determine the location of theeyes of the driver 30. The signals from both sensor systems, 6, 8, 10and 144, are combined in the processor 20, where a determination is madeas to what portions of the mirror should be darkened, e.g., area 143.Appropriate currents are then sent to the mirror 55 in a manner similarto the windshield system described above. Again, an alternative solutionis to observe a glare reflection on the face of the driver and removethe glare with a filter.

Note, the rearview mirror is also an appropriate place to display iconsof the contents of the blind spot or other areas surrounding the vehicleas disclosed in U.S. Pat. No. 7,049,945.

5.3 Visor for Glare Control and HUD

FIG. 24 illustrates the interior of a passenger compartment with a rearview mirror assembly 55, a camera for viewing the eyes of the driver 56and a large generally transparent sun visor 145. The sun visor 145 isnormally largely transparent and is made from electrochromic glass,suspended particle glass, a liquid crystal device or equivalent. Thecamera 56 images the eyes of the driver and looks for a reflectionindicating that glare is impinging on the driver's eyes. The camerasystem may have a source of infrared or other frequency illuminationthat would be momentarily activated to aid in locating the driver'seyes. Once the eyes have been located, the camera monitors the areaaround the eyes, or direct reflections from the eyes themselves, for anindication of glare. The camera system in this case would not know thedirection from which the glare is originating; it would only know thatthe glare was present. The glare blocker system then can darken selectedportions of the visor to attempt to block the source of glare and woulduse the observation of the glare from or around the eyes of the driveras feedback information. When the glare has been eliminated, the systemmaintains the filter, perhaps momentarily reducing it from time to timeto see that the source of glare has not stopped.

If the filter is electrochromic glass, a significant time period isrequired to activate the glare filter and therefore a trial and errorsearch for the ideal filter location could be too slow. In this case, anon-recurring spatial pattern can be placed in the visor such that whenlight passes through the visor and illuminates the face of the driver,the location where the filter should be placed can be easily determined.That is, the pattern reflection off of the face of the driver wouldindicate the location of the visor through which the light causing theglare was passing. Such a structured light system can also be used forthe SPD and LCD filters but since they act significantly more rapidly,it would serve only to simplify the search algorithm for filterplacement.

A second photo sensor 135 can also be used pointing through thewindshield to determine only that glare was present. In this manner,when the source of the glare disappears, the filter can be turned off. Amore sophisticated system as described above for the windshield systemwhereby the direction of the light is determined using a camera-typedevice can also be implemented.

The visor 145 is illustrated as substantially covering the frontwindshield in front of the driver. This is possible since it istransparent except where the filter is applied, which would in generalbe a small area. A second visor, not shown, can also be used to coverthe windshield for the passenger side that would also be useful when thelight-causing glare on the driver's eyes enters thought the windshieldin front of the passenger or if a passenger system is also desired. Insome cases, it might even be advantageous to supply a similar visor tocover the side windows but in general, standard opaque visors wouldserve for both the passenger side windshield area and the side windowssince the driver in general only needs to look through the windshield infront of him or her.

A smaller visor can also be used as long as it is provided with apositioning system or method. The visor only needs to cover the eyes ofthe driver. This could either be done manually or by electric motorssimilar to the system disclosed in U.S. Pat. No. 4,874,938. If electricmotors are used, then the adjustment system would first have to move thevisor so that it covered the driver's eyes and then provide the filter.This could be annoying if the vehicle is heading into the sun andturning and/or going up and down hills. In any case, the visor should bemovable to cover any portion of the windshield where glare can getthrough, unlike conventional visors that only cover the top half of thewindshield. The visor also does not need to be close to the windshieldand the closer that it is to the driver, the smaller and thus the lessexpensive it can be.

As with the windshield, the visor of at least one of the inventionsdisclosed herein can also serve as a display using plastic electronicsas described above either with or without the SPD or other filtermaterial. Additionally, visor-like displays can now be placed at manylocations in the vehicle for the display of Internet web pages, movies,games etc. Occupants of the rear seat, for example, can pull down suchdisplays from the ceiling, up from the front seatbacks or out from theB-pillars or other convenient locations.

A key advantage of the systems disclosed herein is the ability to handlemultiple sources of glare in contrast to the system of U.S. Pat. No.4,874,938, which requires that the multiple sources must be closetogether.

5.4 Headlamp Control

In a similar manner, the forward looking camera(s) can also be used tocontrol the lights of vehicle 136 when either the headlights ortaillights of another vehicle are sensed. In this embodiment, the CCDarray is designed to be sensitive to visible light and a separate sourceof illumination is not used. The key to this technology can be the useof trained pattern recognition algorithms and particularly theartificial neural network. Here, as in the other cases above and inpatents and patent applications referenced above, the patternrecognition system is trained to recognize the pattern of the headlightsof an oncoming vehicle and/or the tail lights of a vehicle in front ofvehicle 136 and to then dim the headlights when either of theseconditions is sensed. It is also trained to not dim the lights for otherreflections such as reflections off of a sign post or the roadway. Oneproblem is to differentiate taillights where dimming is desired fromdistant headlights where dimming is not desired. At least threetechniques can be used: (i) measurement of the spacing of the lightsources, (ii) determination of the location of the light sourcesrelative to the vehicle, and (iii) use of a red filter where thebrightness of the light source through the filter is compared with thebrightness of the unfiltered light. In the case of the taillight, thebrightness of the red filtered and unfiltered light is nearly the samewhile there is a significant difference for the headlight case. In thissituation, either two CCD arrays are used, one with a filter, or afilter which can be removed either electrically, such as with a liquidcrystal, or mechanically. Alternately a fast Fourier transform, or otherspectral analysis technique, of the data can be taken to determine therelative red content.

6. Weight Measurement and Biometrics

One way to determine motion of the occupant(s) is to monitor the weightdistribution of the occupant whereby changes in weight distributionafter an accident would be highly suggestive of movement of theoccupant. A system for determining the weight distribution of theoccupants can be integrated or otherwise arranged in the seats 3 and 4of the vehicle and several patents and publications describe suchsystems. The disclosure in section 6 of U.S. patent application Ser. No.11/558,996 is particularly applicable.

7. Illumination

7.1 Infrared Light

Many forms of illumination can of course be used. Near infrared is apreferred source since it can be produced relatively inexpensively withLEDs and is not seen by vehicle occupants or others outside of thevehicle. The use of spatially modulated (as in structured light) andtemporally modulated (as in amplitude, frequency, pulse, code, random orother such methods) permits additional information to be obtained suchas a three-dimensional image as disclosed by the current assignee inearlier patents. Infrared is also interesting since the human bodynaturally emits IR and this fact can be used to positively identify thatthere is a human occupying a vehicle seat and to determine fairlyaccurately the size of the occupant. This technique only works when theambient temperature is different from body temperature, which is most ofthe time. In some climates, it is possible that the interior temperatureof a vehicle can reach or exceed 100° F., but it is unlikely to stay atthat temperature for long as humans find such a temperatureuncomfortable. However, it is even more unlikely that such a temperaturewill exist except when there is significant natural illumination in thevisible part of the spectrum. Thus, a visual size determination ispossible especially since it is very unlikely that such an occupant willbe wearing heavy or thick clothing. Passive infrared, used of coursewith an imaging system, is thus a viable technique for theidentification of a human occupant if used in conjunction with anoptical system for high temperature situations. Even if the ambienttemperature is nearly the same as body temperature, there will still becontrasts in the image which are sufficient to differentiate an occupantor his or her face from the background. Whereas a single pixel sensor,as in prior art patents to Corrado and Mattes, could give false results,an imaging system such as a focal plane array as disclosed herein canstill operate effectively.

Passive IR is also a good method of finding the eyes and other featuresof the occupant since hair, some hats and other obscuring itemsfrequently do not interfere with the transmission of IR. When active IRillumination is used, the eyes are particularly easy to find due tocorneal reflection and the eyes will be dilated at night when findingthe eyes is most important. Even in glare situations, where the glare iscoming through the windshield, passive IR is particularly useful sinceglass blocks most IR with wavelengths beyond 1.1 microns and thus theglare will not interfere with the imaging of the face.

Particular frequencies of active IR are especially useful for externalmonitoring. Except for monitoring objects close to the vehicle, mostradar systems have a significant divergence angle making imaging morethat a few meters from the vehicle problematic. Thus there is typicallynot enough information from a scene say 100 meters away to permit themonitor to obtain an image that would permit classification of sensedobjects. Using radar, it is difficult to distinguish a car from a truckor a parked car at the side of the road from one on the same lane as thevehicle or from an advertising sign, for example. Normal visual imagingalso will not work in bad weather situations however some frequencies ofIR do penetrate fog, rain and snow sufficiently well as to permit themonitoring of the road at a significant distance and with enoughresolution to permit imaging and thus classification even in thepresence of rain, snow and fog.

As mentioned herein, there are various methods of illuminating theobject or occupant in the passenger compartment. A scanning point of IRcan be used to overcome reflected sunlight. A structured pattern can beused to help achieve a three-dimensional representation of the vehiclecontents. An image can be compared with illumination and without in anattempt to eliminate the effects on natural and uncontrollableillumination. This generally doesn't work very well since the naturalillumination can overpower the IR. Thus it is usually better to developtwo pattern recognition algorithms, one for IR illumination and one fornatural illumination. For the natural illumination case, the entirevisual and near visual spectrum can be used or some subset of it. Forthe case where a rolling shutter is used, the process can be speeded upsubstantially if one line of pixels is subtracted from the adjacent linewhere the illumination is turned on for every other row and off for theintervening rows. In addition to structured light, there are many othermethods of obtaining a 3D image as discussed above.

7.2 Structured Light

In the applications discussed and illustrated above, the source andreceiver of the electromagnetic radiation have frequently been mountedin the same package. This is not necessary and in some implementations,the illumination source will be mounted elsewhere. For example, a laserbeam can be used which is directed along an axis which bisects the anglebetween the center of the seat volume, or other volume of interest, andtwo of the arrays. Such a beam may come from the A-Pillar, for example.The beam, which may be supplemental to the main illumination system,provides a point reflection from the occupying item that, in most cases,can be seen by two receivers, even if they are significantly separatedfrom each other, making it easier to identify corresponding parts in thetwo images. Triangulation thereafter can precisely determination thelocation of the illuminated point. This point can be moved, or a patternof points provided, to provide even more information. In another casewhere it is desired to track the head of the occupant, for example,several such beams can be directed at the occupant's head duringpre-crash braking or even during a crash to provide the fastestinformation as to the location of the head of the occupant for thefastest tracking of the motion of the occupant's head. Since only a fewpixels are involved, even the calculation time is minimized.

In most of the applications above, the assumption has been made thateither a uniform field of light or a scanning spot of light will beprovided. This need not be the case. The light that is emitted ortransmitted to illuminate the object can be structured light. Structuredlight can take many forms starting with, for example, a rectangular orother macroscopic pattern of light and dark that can be superimposed onthe light by passing it through a filter. If a similar pattern isinterposed between the reflections and the camera, a sort ofpseudo-interference pattern can result sometimes known as Moirépatterns. A similar effect can be achieved by polarizing transmittedlight so that different parts of the object that is being illuminatedare illuminated with light of different polarization. Once again, byviewing the reflections through a similarly polarized array, informationcan be obtained as to where the source of light came from which isilluminating a particular object. Any of the transmitter/receiverassemblies or transducers in any of the embodiments above using opticscan be designed to use structured light.

Usually the source of the structured light is displaced eithervertically, laterally or axially from the imager, but this need notnecessarily be the case. One excellent example of the use of structuredlight to determine a 3D image where the source of the structured lightand the imager are on the same axis is illustrated in U.S. Pat. No.5,003,166. Here, the third dimension is obtained by measuring the degreeof blur of the pattern as reflected from the object. This can be donesince the focal point of the structured light is different from thecamera. This is accomplished by projecting it through its own lenssystem and then combining the two paths through the use of a beamsplitter. The use of this or any other form of structured light iswithin the scope of at least one of the inventions disclosed herein.There are so many methods that the details of all of them cannot beenumerated here.

One consideration when using structured light is that the source ofstructured light should not generally be exactly co-located with thearray because in this case, the pattern projected will not change as afunction of the distance between the array and the object and thus thedistance between the array and the object cannot be determined, exceptby the out-of-focus and similar methods discussed above. Thus, it isusually necessary to provide a displacement between the array and thelight source. For example, the light source can surround the array, beon top of the array or on one side of the array. The light source canalso have a different virtual source, i.e., it can appear to come frombehind of the array or in front of the array, a variation of theout-of-focus method discussed above.

For a laterally displaced source of structured light, the goal is todetermine the direction that a particular ray of light had when it wastransmitted from the source. Then, by knowing which pixels wereilluminated by the reflected light ray along with the geometry of thevehicle, the distance to the point of reflection off of the object canbe determined. If a particular light ray, for example, illuminates anobject surface which is near to the source, then the reflection off ofthat surface will illuminate a pixel at a particular point on theimaging array. If the reflection of the same ray however occurs from amore distant surface, then a different pixel will be illuminated in theimaging array. In this manner, the distance from the surface of theobject to the array can be determined by triangulation formulas.Similarly, if a given pixel is illuminated in the imager from areflection of a particular ray of light from the transmitter, andknowing the direction that that ray of light was sent from thetransmitter, then the distance to the object at the point of reflectioncan be determined. If each ray of light is individually recognizable andtherefore can be correlated to the angle at which it was transmitted, afull three-dimensional image can be obtained of the object thatsimplifies the identification problem. This can be done with a singleimager.

One particularly interesting implementation due to its low cost is toproject one or more dots or other simple shapes onto the occupant from aposition which is at an angle relative to the occupant such as 10 to 45degrees from the camera location. These dots will show up as brightspots even in bright sunlight and their location on the image willpermit the position of the occupant to be determined. Since the parts ofthe occupant are all connected with relative accuracy, the position ofthe occupant can now be accurately determined using only one simplecamera. Additionally, the light that makes up the dots can be modulatedand the distance from the dot source can then be determined if there isa receiver at the light source and appropriate circuitry such as usedwith a scanning range meter.

The coding of the light rays coming from the transmitter can beaccomplished in many ways. One method is to polarize the light bypassing the light through a filter whereby the polarization is acombination of the amount and angle of the polarization. This gives twodimensions that can therefore be used to fix the angle that the lightwas sent. Another method is to superimpose an analog or digital signalonto the light which could be done, for example, by using an addressablelight valve, such as a liquid crystal filter, electrochromic filter, or,preferably, a garnet crystal array. Each pixel in this array would becoded such that it could be identified at the imager or other receivingdevice. Any of the modulation schemes could be applied such asfrequency, phase, amplitude, pulse, random or code modulation.

The techniques described above can depend upon either changing thepolarization or using the time, spatial or frequency domains to identifyparticular transmission angles with particular reflections. Spatialpatterns can be imposed on the transmitted light which generally goesunder the heading of structured light. The concept is that if a patternis identifiable, then either the direction of transmitted light can bedetermined or, if the transmission source is co-linear with thereceiver, then the pattern differentially expands or contracts relativeto the field of view as it travels toward the object and then, bydetermining the size or focus of the received pattern, the distance tothe object can be determined. In some cases, Moiré pattern techniquesare utilized.

When the illumination source is not placed on the same axis as thereceiving array, it is typically placed at an angle such as 45 degrees.At least two other techniques can be considered. One is to place theillumination source at 90 degrees to the imager array. In this case,only those surface elements that are closer to the receiving array thanprevious surfaces are illuminated. Thus, significant information can beobtained as to the profile of the object. In fact, if no object isoccupying the seat, then there will be no reflections except from theseat itself. This provides a very powerful technique for determiningwhether the seat is occupied and where the initial surfaces of theoccupying item are located. A combination of the above techniques can beused with temporally or spatially varying illumination. Taking imageswith the same imager but with illumination from different directions canalso greatly enhance the ability to obtain three-dimensionalinformation.

The particular radiation field of the transmitting transducer can alsobe important to some implementations of at least one of the inventionsdisclosed herein. In some techniques, the object which is occupying theseat is the only part of the vehicle which is illuminated. Extreme careis exercised in shaping the field of light such that this is true. Forexample, the objects are illuminated in such a way that reflections fromthe door panel do not occur. Ideally, if only the items which occupy theseat can be illuminated, then the problem of separating the occupantfrom the interior vehicle passenger compartment surfaces can be moreeasily accomplished. Sending illumination from both sides of the vehicleacross the vehicle can accomplish this.

The above discussion has concentrated on automobile occupant sensing butthe teachings, with some modifications, are applicable to monitoring ofother vehicles including railroad cars, truck trailers and cargocontainers.

7.3 Color and Natural Light

As discussed above, the use of multispectral imaging can be asignificant aid in recognizing objects inside and outside of a vehicle.Two objects may not be separable under monochromic illumination yet bequite distinguishable when observed in color or with illumination fromother parts of the electromagnetic spectrum. Also, the identification ofa particular individual is enhanced using near UV radiation, forexample. Even low level X-rays can be useful in identifying and locatingobjects in a vehicle.

7.4 Radar

Particular mention should be made of the use of radar since novelinexpensive antennas and ultra wideband radars are now readilyavailable. A scanning radar beam can be used in this implementation andthe reflected signal is received by a phase array antenna to generate animage of the occupant for input into the appropriate pattern detectioncircuitry. The image is not very clear due to the longer wave lengthsused and the difficulty in getting a small enough radar beam. The wordcircuitry as used herein includes, in addition to normal electroniccircuits, a microprocessor and appropriate software.

Another preferred embodiment makes use of radio waves and avoltage-controlled oscillator (VCO). In this embodiment, the frequencyof the oscillator is controlled through the use of a phase detectorwhich adjusts the oscillator frequency so that exactly one half waveoccupies the distance from the transmitter to the receiver viareflection off of the occupant. The adjusted frequency is thus inverselyproportional to the distance from the transmitter to the occupant.Alternately, an FM phase discriminator can be used as known to thoseskilled in the art. These systems could be used in any of the locationsillustrated in FIG. 5 as well as in the monitoring of other vehicletypes.

In FIG. 6, a motion sensor 73 is arranged to detect motion of anoccupying item on the seat 4 and the output thereof is input to theneural network 65. Motion sensors can utilize a micro-power impulseradar (MIR) system as disclosed, for example, in McEwan U.S. Pat. No.5,361,070, as well as many other patents by the same inventor. Motionsensing is accomplished by monitoring a particular range from the sensoras disclosed in that patent. MIR is one form of radar which hasapplicability to occupant sensing and can be mounted, for example, atlocations such as designated by reference numerals 6 and 8-10 in FIG. 2.It has an advantage over ultrasonic sensors in that data can be acquiredat a higher speed and thus the motion of an occupant can be more easilytracked. The ability to obtain returns over the entire occupancy rangeis somewhat more difficult than with ultrasound resulting in a moreexpensive system overall. MIR has additional advantages over ultrasoundin lack of sensitivity to temperature variation and has a comparableresolution to about 40 kHz ultrasound. Resolution comparable to higherfrequency is feasible but has not been demonstrated. Additionally,multiple MIR sensors can be used when high speed tracking of the motionof an occupant during a crash is required since they can be individuallypulsed without interfering with each, through time divisionmultiplexing. MIR sensors are also particularly applicable to themonitoring of other vehicles and can be configured to provide a systemthat requires very low power and thus is ideal for use withbattery-operated systems that require a very long life.

Sensors 126, 127, 128, 129 in FIG. 21 can also be microwave or mm waveradar sensors which transmit and receive radar waves. As such, it ispossible to determine the presence of an object in the rear seat and thedistance between the object and the sensors. Using multiple radarsensors, it would be possible to determine the contour of an object inthe rear seat and thus using pattern recognition techniques, theclassification or identification of the object. Motion of objects in therear seat can also be determined using radar sensors. For example, ifthe radar sensors are directed toward a particular area and/or areprovided with the ability to detect motion in a predetermined frequencyrange, they can be used to determine the presence of children or petsleft in the vehicle, i.e., by detecting heartbeats or other body motionssuch as movement of the chest cavity.

7.5 Frequency or Spectrum Considerations

The maximum acoustic frequency range that is practical to use foracoustic imaging in the acoustic systems herein is about 40 to 160kilohertz (kHz). The wavelength of a 50 kHz acoustic wave is about 0.6cm, which is too coarse to determine the fine features of a person'sface, for example. It is well understood by those skilled in the artthat features that are smaller than the wavelength of the irradiatingradiation cannot be distinguished. Similarly, the wavelength of commonradar systems varies from about 0.9 cm (for 33 GHz K band) to 133 cm(for 225 MHz P band), which is also too coarse for person identificationsystems. Millimeter wave and sub-millimeter wave radar can of courseemit and receive waves considerably smaller. Millimeter wave radar andMicropower Impulse Radar (MIR) as discussed above are particularlyuseful for occupant detection and especially the motion of occupantssuch as motion caused by heartbeats and breathing, but still too coursefor feature identification. For security purposes, for example, MIR canbe used to detect the presence of weapons on a person that might beapproaching a vehicle such as a bus, truck or train and thus provide awarning of a potential terrorist threat. Passive IR is also useful forthis purpose.

MIR is reflected by edges, joints and boundaries and through thetechnique of range gating, particular slices in space can be observed.Millimeter wave radar, particularly in the passive mode, can also beused to locate life forms because they naturally emit waves atparticular wave lengths such as 3 mm. A passive image of such a personwill also show the presence of concealed weapons as they block thisradiation. Similarly, active millimeter wave radar reflects off ofmetallic objects but is absorbed by the water in a life form. Theabsorption property can be used by placing a radar receiver or reflectorbehind the occupant and measuring the shadow caused by the absorption.The reflective property of weapons including plastics can be used asabove to detect possible terrorist threats. Finally, the use ofsub-millimeter waves again using a detector or reflector on the otherside of the occupant can be used not only to determine the density ofthe occupant but also some measure of its chemical composition as thechemical properties alter the pulse shape. Such waves are more readilyabsorbed by water than by plastic. From the above discussion, it can beseen that there are advantages of using different frequencies of radarfor different purposes and, in some cases, a combination of frequenciesis most useful. This combination occurs naturally with noise radar (NR),ultra-wideband radar (UWB) and MIR and these technologies are mostappropriate for occupant detection when using electromagnetic radiationat longer wavelengths than visible light and IR.

Another variant on the invention is to use no illumination source atall. In this case, the entire visible and infrared spectrum could beused. CMOS arrays are now available with very good night visioncapabilities making it possible to see and image an occupant in very lowlight conditions. QWIP, as discussed above, may someday become availablewhen on-chip cooling systems using a dual stage Peltier system becomecost effective or when the operating temperature of the device risesthrough technological innovation. For a comprehensive introduction tomultispectral imaging, see Richards, Austin Alien Vision, Exploring theElectromagnetic Spectrum with Imaging Technology, SPIE Press, 2001.

Thus many different frequencies can be used to image a scene each havingparticular advantages and disadvantages. At least one of the inventionsdisclosed herein is not limited to using a particular frequency or partof the electromagnetic spectrum and images can advantageously becombined from different frequencies. For example, a radar image can becombined or fused with an image from the infrared or ultravioletportions of the spectrum. Additionally, the use of a swept frequencyrange such as in a chirp can be advantageously used to distinguishdifferent objects or in some cases different materials. It is well knownthat different materials absorb and reflect different electromagneticwaves and that this fact can be used to identify the material as inspectrographic analysis.

8. Field Sensors and Antennas

A living object such as an animal or human has a fairly high electricalpermittivity (Dielectric Constant) and relatively lossy dielectricproperties (Loss Tangent) absorbs a lot of energy absorption when placedin an appropriate varying electric field. This effect varies with thefrequency. If a human, which is a lossy dielectric, is present in thedetection field, then the dielectric absorption causes the value of thecapacitance of the object to change with frequency. For a human (poordielectric) with high dielectric losses (loss tangent), the decay withfrequency will be more pronounced than objects that do not present thishigh loss tangency. Exploiting this phenomena, it is possible to detectthe presence of an adult, child, baby or pet that is in the field of thedetection circuit.

In FIG. 6, a capacitive sensor 78 is arranged to detect the presence ofan occupying item on the seat 4 and the output thereof is input to theneural network 65. Capacitive sensors can be located many other placesin the passenger compartment. Capacitive sensors appropriate for thisfunction are disclosed in U.S. Pat. Nos. 5,602,734, 5,802,479, 5,844,486and 5,948,031. Capacitive sensors can in general be mounted at locationsdesignated by reference numerals 6 and 8-10 in FIG. 2 or as shown inFIG. 6 or in the vehicle seat and seatback, although by their naturethey can occupy considerably more space than shown in the drawings.

In FIG. 4, transducers 5, 11, 12, 13, 14 and 15 can be antennas placedin the seat and headrest such that the presence of an object,particularly a water-containing object such as a human, disturbs thenear field of the antenna. This disturbance can be detected by variousmeans such as with Micrel parts MICREF102 and MICREF104, which have abuilt-in antenna auto-tune circuit. Note, these parts cannot be used asis and it is necessary to redesign the chips to allow the auto-tuneinformation to be retrieved from the chip.

Note that the bio-impedance that can be measured using the methodsdescribed above can be used to obtain a measure of the water mass, forexample, of an object and thus of its weight.

9. Telematics

Some of the inventions herein relate generally to telematics and thetransmission of information from a vehicle to one or more remote siteswhich can react to the position or status of the vehicle and/oroccupant(s) therein. Details of the manner in which telematics can beapplied to the invention are descried in U.S. patent application Ser.No. 11/558,996.

10. Display

A portion of the windshield, such as the lower left corner, can be usedto display the vehicle and surrounding vehicles or other objects as seenfrom above, for example, as described in U.S. Pat. No. 7,049,945. Thisdisplay can use pictures or icons as appropriate. In another case, thecondition of the road such as the presence, or likelihood of black icecan be displayed on the windshield where it would show on the road ifthe driver could see it. This would require a source of information thatsuch a condition exists, however, here the concern is that it can bedisplayed whatever the source of this or any other relevant information.When used in conjunction with a navigation system, directions includingpointing arrows or a path outline perhaps in color, similar to the firstdown line on a football field as seen on TV, can be displayed to directthe driver to his destination or to points of interest.

11. Pattern Recognition

In basic embodiments of the inventions, wave or energy-receivingtransducers are arranged in the vehicle at appropriate locations,associated algorithms are trained, if necessary depending on theparticular embodiment, and function to determine whether a life form, orother object, is present in the vehicle and if so, how many life formsor objects are present. A determination can also be made using thetransducers as to whether the life forms are humans, or morespecifically, adults, child in child seats, etc. As noted above andbelow, this is possible using pattern recognition techniques. Moreover,the processor or processors associated with the transducers can betrained (loaded with a trained pattern recognition algorithm) todetermine the location of the life forms or objects, either periodicallyor continuously or possibly only immediately before, during and after acrash. The location of the life forms or objects can be as general or asspecific as necessary depending on the system requirements, i.e., adetermination can be made that a human is situated on the driver's seatin a normal position (general) or a determination can be made that ahuman is situated on the driver's seat and is leaning forward and/or tothe side at a specific angle as well as determining the position of hisor her extremities and head and chest (specific). Or, a determinationcan be made as to the size or type of objects such as boxes are in atruck trailer or cargo container. The degree of detail is limited byseveral factors, including, e.g., the number, position and type oftransducers and the training of the pattern recognition algorithm.

When different objects are placed on the front passenger seat, theimages (here “image” is used to represent any form of signal) fromtransducers 6, 8, 10 (FIG. 1) are different for different objects butthere are also similarities between all images of rear facing childseats, for example, regardless of where on the vehicle seat it is placedand regardless of what company manufactured the child seat. Alternately,there will be similarities between all images of people sitting on theseat regardless of what they are wearing, their age or size. The problemis to find the set of “rules” or an algorithm that differentiates theimages of one type of object from the images of other types of objects,for example which differentiate the adult occupant images from the rearfacing child seat images or boxes. The similarities of these images forvarious child seats are frequently not obvious to a person looking atplots of the time series from ultrasonic sensors, for example, and thuscomputer algorithms are developed to sort out the various patterns. Fora more detailed discussion of pattern recognition, see U.S. RE37260.

The determination of these rules is important to the pattern recognitiontechniques used in at least one of the inventions disclosed herein. Ingeneral, three approaches have been useful, artificial intelligence,fuzzy logic and artificial neural networks including modular orcombination neural networks. Other types of pattern recognitiontechniques may also be used, such as sensor fusion as disclosed in U.S.Pat. Nos. 5,482,314, 5,890,085, and 6,249,729. In some of the inventionsdisclosed herein, such as the determination that there is an object inthe path of a closing window or door using acoustics or optics asdescribed herein, the rules are sufficiently obvious that a trainedresearcher can look at the returned signals and devise an algorithm tomake the required determinations. In others, such as the determinationof the presence of a rear facing child seat or of an occupant,artificial neural networks are used to determine the rules. Neuralnetwork software for determining the pattern recognition rules isavailable from various sources such as International ScientificResearch, Inc., Panama City, Panama.

The human mind has little problem recognizing faces even when they arepartially occluded such as with a hat, sunglasses or a scarf, forexample. With the increase in low cost computing power, it is nowbecoming possible to train a rather large neural network, perhaps acombination neural network, to recognize most of those cases where ahuman mind will also be successful.

Other techniques which may or may not be part of the process ofdesigning a system for a particular application include the following:

1. Fuzzy logic. Neural networks frequently exhibit the property thatwhen presented with a situation that is totally different from anypreviously encountered, an irrational decision can result. Frequently,when the trained observer looks at input data, certain boundaries to thedata become evident and cases that fall outside of those boundaries areindicative of either corrupted data or data from a totally unexpectedsituation. It is sometimes desirable for the system designer to addrules to handle these cases. These can be fuzzy logic-based rules orrules based on human intelligence. One example would be that whencertain parts of the data vector fall outside of expected bounds thatthe system defaults to an airbag-enable state or the previouslydetermined state.

2. Genetic algorithms. When developing a neural network algorithm for aparticular vehicle, there is no guarantee that the best of all possiblealgorithms has been selected. One method of improving the probabilitythat the best algorithm has been selected is to incorporate some of theprinciples of genetic algorithms. In one application of this theory, thenetwork architecture and/or the node weights are varied pseudo-randomlyto attempt to find other combinations which have higher success rates.The discussion of such genetic algorithms systems appears in the bookComputational Intelligence referenced above.

Although neural networks are preferred other classifiers such asBayesian classifiers can be used as well as any other patternrecognition system. A key feature of most of the inventions disclosedherein is the recognition that the technology of pattern recognitionrather than deterministic mathematics should be applied to solving theoccupant sensing problem.

11.1 Neural Networks

An occupant can move from a position safely displaced from the airbag toa position where he or she can be seriously injured by the deployment ofan airbag within a fraction of a second during pre-crash braking, forexample. On the other hand, it takes a substantially longer time periodto change the seat occupancy state from a forward facing person to arear facing child seat, or even from a forward facing child seat to arear facing child seat. This fact can be used in the discriminationprocess through post-processing algorithms. One method, which alsoprepares for DOOP, is to use a two-layered neural network or twoseparate neural networks. The first one categorizes the seat occupancyinto, for example, (1) empty seat, (2) rear facing child seat, (3)forward facing child seat and (4) forward facing human (not in a childseat). The second is used for occupant position determination. In theimplementation, the same input layer can be used for both neuralnetworks but separate hidden and output layers are used. This isillustrated in FIG. 187 of the '881 application which is similar to FIG.19B with the addition of a post processing operation for both thecategorization and position networks and the separate hidden layer nodesfor each network.

If the categorization network determines that either a category (3) or(4) exists, then the second network is run, which determines thelocation of the occupant. Significant averaging of the vectors is usedfor the first network and substantial evidence is required before theoccupancy class is changed. For example, if data is acquired every 10milliseconds, the first network might be designed to require 600 out of1000 changed vectors before a change of state is determined. In thiscase, at least 6 seconds of confirming data would be required. Such asystem would therefore not be fooled by a momentary placement of anewspaper by a forward facing human, for example, that might look like arear-facing child seat.

If, on the other hand, a forward facing human were chosen, his or herposition could be determined every 10 milliseconds. A decision that theoccupant had moved out of position would not necessarily be made fromone 10 millisecond reading unless that reading was consistent withprevious readings. Nevertheless, a series of consistent readings wouldlead to a decision within 10 milliseconds of when the occupant crossedover into the danger zone proximate to the airbag module. This method ofusing history is used to eliminate the effects of temperature gradients,for example, or other events that could temporarily distort one or morevectors. The algorithms which perform this analysis are part of thepost-processor.

More particularly, in one embodiment of the method in accordance with atleast one of the inventions herein in which two neural networks are usedin the control of the deployment of an occupant restraint device basedon the position of an object in a passenger compartment of a vehicle,several wave-emitting and receiving transducers are mounted on thevehicle. In one preferred embodiment, the transducers are ultrasonictransducers which simultaneously transmit and receive waves at differentfrequencies from one another. A determination is made by a first neuralnetwork whether the object is of a type requiring deployment of theoccupant restraint device in the event of a crash involving the vehiclebased on the waves received by at least some of the transducers afterbeing modified by passing through the passenger compartment. If so,another determination is made by a second neural network whether theposition of the object relative to the occupant restraint device wouldcause injury to the object upon deployment of the occupant restraintdevice based on the waves received by at least some of the transducers.The first neural network is trained on signals from at least some of thetransducers representative of waves received by the transducers whendifferent objects are situated in the passenger compartment. The secondneural network is trained on signals from at least some of thetransducers when different objects in different positions are situatedin the passenger compartment.

The transducers used in the training of the first and second neuralnetworks and operational use of method are not necessary the sametransducers and different sets of transducers can be used for the typingor categorizing of the object via the first neural network and theposition determination of the object via the second neural network.

The modifications described above with respect to the use of ultrasonictransducers can also be used in conjunction with a dual neural networksystem. For example, motion of a respective vibrating element or cone ofone or more of the transducers may be electronically or mechanicallydiminished or suppressed to reduce ringing of the transducer and/or oneor more of the transducers may be arranged in a respective tube havingan opening through which the waves are transmitted and received.

In another embodiment of the invention, a method for categorizing anddetermining the position of an object in a passenger compartment of avehicle entails mounting a plurality of wave-receiving transducers onthe vehicle, training a first neural network on signals from at leastsome of the transducers representative of waves received by thetransducers when different objects in different positions are situatedin the passenger compartment, and training a second neural network onsignals from at least some of the transducers representative of wavesreceived by the transducers when different objects in differentpositions are situated in the passenger compartment. As such, the firstneural network provides an output signal indicative of thecategorization of the object while the second neural network provides anoutput signal indicative of the position of the object. The transducersmay be controlled to transmit and receive waves each at a differentfrequency, as discussed herein, and one or more of the transducers maybe arranged in a respective tube having an opening through which thewaves are transmitted and received.

Although this system is described with particular advantageous use forultrasonic and optical transducers, it is conceivable that othertransducers other than the ultrasonics or optics can also be used inaccordance with the invention. A dual neural network is a form of amodular neural network and both are subsets of combination neuralnetworks.

The system used in a preferred implementation of at least one of theinventions disclosed herein for the determination of the presence of arear facing child seat, of an occupant or of an empty seat, for example,is the artificial neural network, which is also commonly referred to asa trained neural network. In one case, illustrated in FIG. 1, thenetwork operates on the returned signals as sensed by transducers 6, 8,9 and 10, for example. Through a training session, the system is taughtto differentiate between the different cases. This is done by conductinga large number of experiments where a selection of the possible childseats is placed in a large number of possible orientations on the frontpassenger seat. Similarly, a sufficiently large number of experimentsare run with human occupants and with boxes, bags of groceries and otherobjects (both inanimate and animate). For each experiment with differentobjects and the same object in different positions, the returned signalsfrom the transducers 6, 8, 9 and 10, for example, are associated withthe identification of the occupant in the seat or the empty seat andinformation about the occupant such as its orientation if it is a childseat and/or position. Data sets are formed from the returned signals andthe identification and information about the occupant or the absence ofan occupant. The data sets are input into a neural network-generatingprogram that creates a trained neural network that can, upon receivinginput of returned signals from the transducers 6, 8, 9 and 10, providean output of the identification and information about the occupant mostlikely situated in the seat or ascertained the existence of an emptyseat. Sometimes as many as 1,000,000 such experiments are run before theneural network is sufficiently trained and tested so that it candifferentiate among the several cases and output the correct decisionwith a very high probability. The data from each trial is combined toform a one-dimensional array of data called a vector. Of course, it mustbe realized that a neural network can also be trained to differentiateamong additional cases, for example, a forward facing child seat. It canalso be trained to recognize the existence of one or more boxes or othercargo within a truck trailer, cargo container, automobile trunk orrailroad car, for example.

Considering now FIG. 9, the normalized data from the ultrasonictransducers 6, 8, 9 and 10, the seat track position detecting sensor 74,the reclining angle detecting sensor 57, from the weight sensor(s) 7, 76and 97, from the heartbeat sensor 71, the capacitive sensor 78 and themotion sensor 73 are input to the neural network 65, and the neuralnetwork 65 is then trained on this data. More specifically, the neuralnetwork 65 adds up the normalized data from the ultrasonic transducers,from the seat track position detecting sensor 74, from the recliningangle detecting sensor 57, from the weight sensor(s) 7, 76 and 97, fromthe heartbeat sensor 71, from the capacitive sensor 78 and from themotion sensor 73 with each data point multiplied by an associated weightaccording to the conventional neural network process to determinecorrelation function (step S6 in FIG. 15).

Looking now at FIG. 19B, in this embodiment, 144 data points areappropriately interconnected at 25 connecting points of layer 1, andeach data point is mutually correlated through the neural networktraining and weight determination process. The 144 data points consistof 138 measured data points from the ultrasonic transducers, the data(139th) from the seat track position detecting sensor 74, the data(140th) from the reclining angle detecting sensor 57, the data (141st)from the weight sensor(s) 7 or 76, the data (142^(nd)) from theheartbeat sensor 71, the data (143^(rd)) from the capacitive sensor andthe data (144^(th)) from the motion sensor (the last three inputs arenot shown on FIG. 19B. Each of the connecting points of the layer 1 hasan appropriate threshold value, and if the sum of measured data exceedsthe threshold value, each of the connecting points will output a signalto the connecting points of layer 2. Although the weight sensor input isshown as a single input, in general there will be a separate input fromeach weight sensor used. For example, if the seat has four seat supportsand a strain measuring element is used on each support, what will befour data inputs to the neural network.

The connecting points of the layer 2 comprises 20 points, and the 25connecting points of the layer 1 are appropriately interconnected as theconnecting points of the layer 2. Similarly, each data is mutuallycorrelated through the training process and weight determination asdescribed above and in above-referenced neural network texts. Each ofthe 20 connecting points of the layer 2 has an appropriate thresholdvalue, and if the sum of measured data exceeds the threshold value, eachof the connecting points will output a signal to the connecting pointsof layer 3.

The connecting points of the layer 3 comprises 3 points, and theconnecting points of the layer 2 are interconnected at the connectingpoints of the layer 3 so that each data is mutually correlated asdescribed above. If the sum of the outputs of the connecting points oflayer 2 exceeds a threshold value, the connecting points of the latter 3will output Logic values (100), (010), and (001) respectively, forexample.

The neural network 65 recognizes the seated-state of a passenger A bytraining as described in several books on Neural Networks mentioned inabove referenced patents and patent applications. Then, after trainingthe seated-state of the passenger A and developing the neural networkweights, the system is tested. The training procedure and the testprocedure of the neural network 65 will hereafter be described with aflowchart shown in FIG. 15.

The threshold value of each connecting point is determined bymultiplying weight coefficients and summing up the results in sequence,and the aforementioned training process is to determine a weightcoefficient Wj so that the threshold value (ai) is a previouslydetermined output.

ai=ΣWj·Xj (j=1 to N)

wherein

-   -   Wj is the weight coefficient,    -   Xj is the data and    -   N is the number of samples.

Based on this result of the training, the neural network 65 generatesthe weights for the coefficients of the correlation function or thealgorithm (step S7).

At the time the neural network 65 has learned a suitable number ofpatterns of the training data, the result of the training is tested bythe test data. In the case where the rate of correct answers of theseated-state detecting unit based on this test data is unsatisfactory,the neural network is further trained and the test is repeated. In thisembodiment, the test was performed based on about 600,000 test patterns.When the rate of correct test result answers was at about 98%, thetraining was ended. Further improvements to the ultrasonic occupantsensor system has now resulted in accuracies exceeding 98% and for theoptical system exceeding 99%.

The neural network software operates as follows. The training data isused to determine the weights which multiply the values at the variousnodes at the lower level when they are combined at nodes at a higherlevel. Once a sufficient number of iterations have been accomplished,the independent data is used to check the network. If the accuracy ofthe network using the independent data is lower than the last time thatit was checked using the independent data, then the previous weights aresubstituted for the new weights and training of the network continues ona different path. Thus, although the independent data is not used totrain the network, it does strongly affect the weights. It is thereforenot really independent. Also, both the training data and the independentdata are created so that all occupancy states are roughly equallyrepresented. As a result, a third set of data is used which isstructured to more closely represent the real world of vehicleoccupancy. This third data set, the “real world” data, is then used toarrive at a figure as to the real accuracy of the system.

The neural network 65 has outputs 65 a, 65 b and 65 c (FIG. 9). Each ofthe outputs 65 a, 65 b and 65 c outputs a signal of logic 0 or 1 to agate circuit or algorithm 77. Based on the signals from the outputs 65a, 65 b and 65 c, any one of these combination (100), (010) and (001) isobtained. In another preferred embodiment, all data for the empty seatwas removed from the training set and the empty seat case was determinedbased on the output of the weight sensor alone. This simplifies theneural network and improves its accuracy.

In this embodiment, the output (001) correspond to a vacant seat, a seatoccupied by an inanimate object or a seat occupied by a pet (VACANT),the output (010) corresponds to a rear facing child seat (RFCS) or anabnormally seated passenger (ASP or OOPA), and the output (100)corresponds to a normally seated passenger (NSP or FFA) or a forwardfacing child seat (FFCS).

The gate circuit (seated-state evaluation circuit) 77 can be implementedby an electronic circuit or by a computer algorithm by those skilled inthe art and the details will not be presented here. The function of thegate circuit 77 is to remove the ambiguity that sometimes results whenultrasonic sensors and seat position sensors alone are used. Thisambiguity is that it is sometimes difficult to differentiate between arear facing child seat (RFCS) and an abnormally seated passenger (ASP),or between a normally seated passenger (NSP) and a forward facing childseat (FFCS). By the addition of one or more weight sensors in thefunction of acting as a switch when the weight is above or below 60lbs., it has been found that this ambiguity can be eliminated. The gatecircuit therefore takes into account the output of the neural networkand also the weight from the weight sensor(s) as being above or below 60lbs. and thereby separates the two cases just described and results infive discrete outputs.

The use of weight data must be heavily filtered since during drivingconditions, especially on rough roads or during an accident, the weightsensors will give highly varying output. The weight sensors, therefore,are of little value during the period of time leading up to andincluding a crash and their influence must be minimized during this timeperiod. One way of doing this is to average the data over a long periodof time such as from 5 seconds to a minute or more.

Thus, the gate circuit 77 fulfills a role of outputting five kinds ofseated-state evaluation signals, based on a combination of three kindsof evaluation signals from the neural network 65 and superimposedinformation from the weight sensor(s). The five seated-state evaluationsignals are input to an airbag deployment determining circuit that ispart of the airbag system and will not be described here. As disclosedin above-referenced patents and patent applications, output of thissystem can also be used to activate a variety of lights or alarms toindicate to the operator of the vehicle the seated state of thepassenger. The system that has been described here for the passengerside is also applicable for the most part for the driver side.

An alternate and preferred method of accomplishing the functionperformed by the gate circuit is to use a modular neural network. Inthis case, the first level neural network is trained on determiningwhether the seat is occupied or vacant. The input to this neural networkconsists of all of the data points described above. Since the onlyfunction of this neural network is to ascertain occupancy, the accuracyof this neural network is very high. If this neural network determinesthat the seat is not vacant, then the second level neural networkdetermines the occupancy state of the seat.

In this embodiment, although the neural network 65 has been employed asan evaluation circuit, the mapping data of the coefficients of acorrelation function may also be implemented or transferred to amicrocomputer to constitute the evaluation circuit (see Step S8 in FIG.15).

According to the seated-state detecting unit of the present invention,the identification of a vacant seat (VACANT), a rear facing child seat(RFCS), a forward facing child seat (FFCS), a normally seated adultpassenger (NSP), an abnormally seated adult passenger (ASP), can bereliably performed. Based on this identification, it is possible tocontrol a component, system or subsystem in the vehicle. For example, aregulation valve which controls the inflation or deflation of an airbagmay be controlled based on the evaluated identification of the occupantof the seat. This regulation valve may be of the digital or analog type.A digital regulation valve is one that is in either of two states, openor closed. The control of the flow is then accomplished by varying thetime that the valve is open and closed, i.e., the duty cycle.

The neural network has been previously trained on a significant numberof occupants of the passenger compartment. The number of such occupantsdepends strongly on whether the driver or the passenger seat is beinganalyzed. The variety of seating states or occupancies of the passengerseat is vastly greater than that of the driver seat. For the driverseat, a typical training set will consist of approximately 100 differentvehicle occupancies. For the passenger seat, this number can exceed1000. These numbers are used for illustration purposes only and willdiffer significantly from vehicle model to vehicle model. Of course manyvectors of data will be taken for each occupancy as the occupant assumesdifferent positions and postures.

The neural network is now used to determine which of the storedoccupancies most closely corresponds to the measured data. The output ofthe neural network can be an index of the setup that was used duringtraining that most closely matches the current measured state. Thisindex can be used to locate stored information from the matched trainedoccupancy. Information that has been stored for the trained occupancytypically includes the locus of the centers of the chest and head of thedriver, as well as the approximate radius of pixels which is associatedwith this center to define the head area, for example. For the case ofFIG. 8A, it is now known from this exercise where the head, chest, andperhaps the eyes and ears, of the driver are most likely to be locatedand also which pixels should be tracked in order to know the preciseposition of the driver's head and chest. What has been described aboveis the identification process for automobile occupancy and is onlyrepresentative of the general process. A similar procedure, althoughusually simpler with fewer steps, is applicable to other vehiclemonitoring cases.

The use of trainable pattern recognition technologies such as neuralnetworks is an important part of the some of the inventions disclosesherein particularly for the automobile occupancy case, although othernon-trained pattern recognition systems such as fuzzy logic,correlation, Kalman filters, and sensor fusion can also be used. Thesetechnologies are implemented using computer programs to analyze thepatterns of examples to determine the differences between differentcategories of objects. These computer programs are derived using a setof representative data collected during the training phase, called thetraining set. After training, the computer programs output a computeralgorithm containing the rules permitting classification of the objectsof interest based on the data obtained after installation in thevehicle. These rules, in the form of an algorithm, are implemented inthe system that is mounted onto the vehicle. The determination of theserules is important to the pattern recognition techniques used in atleast one of the inventions disclosed herein. Artificial neural networksusing back propagation are thus far the most successful of the ruledetermination approaches, however, research is underway to developsystems with many of the advantages of back propagation neural networks,such as learning by training, without the disadvantages, such as theinability to understand the network and the possibility of notconverging to the best solution. In particular, back propagation neuralnetworks will frequently give an unreasonable response when presentedwith data than is not within the training data. It is well known thatneural networks are good at interpolation but poor at extrapolation. Acombined neural network fuzzy logic system, on the other hand, cansubstantially solve this problem. Additionally, there are many otherneural network systems in addition to back propagation. In fact, onetype of neural network may be optimum for identifying the contents ofthe passenger compartment and another for determining the location ofthe object dynamically.

Numerous books and articles, including more that 500 U.S. patents,describe neural networks in great detail and thus the theory andapplication of this technology is well known and will not be repeatedhere. Except in a few isolated situations where neural networks havebeen used to solve particular problems limited to engine control, forexample, they have not previously been applied to automobiles, trucks orother vehicle monitoring situations.

The system generally used in the instant invention, therefore, for thedetermination of the presence of a rear facing child seat, an occupant,or an empty seat is the artificial neural network or a neural-fuzzysystem. In this case, the network operates on the returned signals froma CCD or CMOS array as sensed by transducers 49, 50, 51 and 54 in FIG.8D, for example. For the case of the front passenger seat, for example,through a training session, the system is taught to differentiatebetween the three cases. This is done by conducting a large number ofexperiments where available child seats are placed in numerous positionsand orientations on the front passenger seat of the vehicle.

Once the network is determined, it is possible to examine the result todetermine, from the algorithm created by the neural network software,the rules that were finally arrived at by the trial and error trainingtechnique. In that case, the rules can then be programmed into amicroprocessor. Alternately, a neural computer can be used to implementthe neural network directly. In either case, the implementation can becarried out by those skilled in the art of pattern recognition usingneural networks. If a microprocessor is used, a memory device is alsorequired to store the data from the analog to digital converters whichdigitize the data from the receiving transducers. On the other hand, ifa neural network computer is used, the analog signal can be fed directlyfrom the transducers to the neural network input nodes and anintermediate memory is not required. Memory of some type is needed tostore the computer programs in the case of the microprocessor system andif the neural computer is used for more than one task, a memory isneeded to store the network specific values associated with each task.

A review of the literature on neural networks yields the conclusion thatthe use of such a large training set is unique in the neural networkfield. The rule of thumb for neural networks is that there must be atleast three training cases for each network weight. Thus, for example,if a neural network has 156 input nodes, 10 first hidden layer nodes, 5second hidden layer nodes, and one output node this results in a totalof 1,622 weights. According to conventional theory 5000 trainingexamples should be sufficient. It is highly unexpected, therefore, thatgreater accuracy would be achieved through 100 times that many cases. Itis thus not obvious and cannot be deduced from the neural networkliterature that the accuracy of the system will improve substantially asthe size of the training database increases even to tens of thousands ofcases. It is also not obvious looking at the plots of the vectorsobtained using ultrasonic transducers that increasing the number oftests or the database size will have such a significant effect on thesystem accuracy. Each of the vectors is typically a rather course plotwith a few significant peaks and valleys. Since the spatial resolutionof an ultrasonic system is typically about 2 to 4 inches, it is onceagain surprising that such a large database is required to achievesignificant accuracy improvements.

The back propagation neural network is a very successful general-purposenetwork. However, for some applications, there are other neural networkarchitectures that can perform better. If it has been found, forexample, that a parallel network as described above results in asignificant improvement in the system, then, it is likely that theparticular neural network architecture chosen has not been successful inretrieving all of the information that is present in the data. In such acase, an RCE, Stochastic, Logicon Projection, cellular, support vectormachine or one of the other approximately 30 types of neural networkarchitectures can be tried to see if the results improve. This parallelnetwork test, therefore, is a valuable tool for determining the degreeto which the current neural network is capable of using efficiently theavailable data.

One of the salient features of neural networks is their ability of findpatterns in data regardless of its source. Neural networks work wellwith data from ultrasonic sensors, optical imagers, strain gage andbladder weight sensors, temperature sensors, chemical sensors, radiationsensors, pressure sensors, electric field sensors, capacitance basedsensors, any other wave sensors including the entire electromagneticspectrum, etc. If data from any sensors can be digitized and fed into aneural network generating program and if there is information in thepattern of the data then neural networks can be a viable method ofidentifying those patterns and correlating them with a desired outputfunction. Note that although the inventions disclosed herein preferablyuse neural networks and combination neural networks to be describednext, these inventions are not limited to this form or method of patternrecognition. The major breakthrough in occupant sensing came with therecognition by the current assignee that ordinary analysis usingmathematical equations where the researcher looks at the data andattempts, based on the principles of statistics, engineering or physics,to derive the relevant relationships between the data and the categoryand location of an occupying item, is not the proper approach and thatpattern recognition technologies should be used. This is believed to bethe first use of such pattern recognition technologies in the automobilesafety and monitoring fields with the exception that neural networkshave been used by the current assignee and others as the basis of acrash sensor algorithm and by certain automobile manufacturers forengine control. Note for many monitoring situations in truck trailers,cargo containers and railroad cars where questions such as “is thereanything in the vehicle?” are asked, neural networks may not always berequired.

11.2 Combination Neural Networks

The technique described above for the determination of the location ofan occupant during panic or braking pre-crash situations involves use ofa modular neural network. In that case, one neural network was used todetermine the occupancy state of the vehicle and one or more neuralnetworks were used to determine the location of the occupant within thevehicle. The method of designing a system utilizing multiple neuralnetworks is a key teaching of the present invention. When this idea isgeneralized, many potential combinations of multiple neural networkarchitectures become possible. Some of these are discussed in U.S.patent application Ser. No. 11/558,996.

11.3 Interpretation of Other Occupant States

Once a vehicle interior monitoring system employing a sophisticatedpattern recognition system, such as a neural network or modular neuralnetwork, is in place, it is possible to monitor the motions of thedriver over time and determine if he is falling asleep or has otherwisebecome incapacitated. In such an event, the vehicle can be caused torespond in a number of different ways. One such system is illustrated inFIG. 6 and consists of a monitoring system having transducers 8 and 9plus microprocessor 20 programmed to compare the motions of the driverover time and trained to recognize changes in behavior representative ofbecoming incapacitated e.g., the eyes blinking erratically and remainingclosed for ever longer periods of time. If the system determines thatthere is a reasonable probability that the driver has fallen asleep, forexample, then it can turn on a warning light shown here as 41 or send awarning sound. If the driver fails to respond to the warning by pushinga button 43, for example, then the horn and lights can be operated in amanner to warn other vehicles and the vehicle brought to a stop. Onenovel approach, not shown, would be to use the horn as the button 43.For a momentary depression of the horn, for this case, the horn wouldnot sound. Other responses can also be programmed and other tests ofdriver attentiveness can be used, without resorting to attempting tomonitor the motions of the driver's eyes that would signify that thedriver was alert. These other responses can include an input to thesteering wheel, motion of the head, blinking or other motion of the eyesetc. In fact, by testing a large representative sample of the populationof drivers, the range of alert responses to the warning light and/orsound can be compared to the lack of response of a sleeping driver andthereby the state of attentiveness determined.

An even more sophisticated system of monitoring the behavior of thedriver is to track his eye motions using such techniques as aredescribed in U.S. Pat. Nos. 4,648,052, 4,720,189, 4,836,670, 4,950,069,5,008,946 and 5,305,012. Detection of the impaired driver in particularcan be best determined by these techniques. These systems use patternrecognition techniques plus, in many cases, the transmitter and CCDreceivers must be appropriately located so that the reflection off ofthe cornea of the driver's eyes can be detected as discussed inabove-referenced patents. The size of the CCD arrays used herein permitstheir location, sometimes in conjunction with a reflective windshield,where this corneal reflection can be detected with some difficulty.Sunglasses or other items can interfere with this process.

In a similar manner as described in these patents, the motion of thedriver's eyes can be used to control various systems in the vehiclepermitting hands off control of the entertainment system, heating andair conditioning system or all of the other systems described above.Although some of these systems have been described in theafore-mentioned patents, none have made use of neural networks forinterpreting the eye movements. The use of particular IR wavelengthspermits the monitoring of the driver's eyes without the driver knowingthat this is occurring. IR with a wave length above about 1.1 microns,however, is blocked by glass eyeglasses and thus other invisiblefrequencies may be required.

The use of the windshield as a reflector is particularly useful whenmonitoring the eyes of the driver by means of a camera mounted on therear view mirror assembly. The reflections from the cornea are highlydirectional, as every driver knows whose lights have reflected off theeyes of an animal on the roadway. For this to be effective, the eyes ofthe driver must be looking at the radiation source. Since the driver ispresumably looking through the windshield, the source of the radiationmust also come from the windshield and the reflections from the driver'seyes must also be in the direction of the windshield. Using thistechnique, the time that the driver spends looking through thewindshield can be monitored and if that time drops below some thresholdvalue, it can be presumed that the driver is not attentive and may besleeping or otherwise incapacitated.

The location of the eyes of the driver, for this application, is greatlyfacilitated by the teachings of the inventions as described above.Although others have suggested the use of eye motions and cornealreflections for drowsiness determination, up until now there has notbeen a practical method for locating the driver's eyes with sufficientprecision and reliability as to render this technique practical. Also,although sunglasses might defeat such a system, most drowsiness causedaccidents happen at night when it is less likely that sunglasses areworn.

11.4 Combining Occupant Monitoring and Car Monitoring

There is an inertial measurement unit (IMU) under development by thecurrent assignee that will have the equivalent accuracy as an expensivemilitary IMU but will sell for under $200 in sufficient volume. This IMUcan contain three accelerometers and three gyroscopes and permit a veryaccurate tracking of the motion of the vehicle in three dimensions. Themain purposes of this device will be replace all non-crush zone crashand rollover sensors, chassis control gyros etc. with a single devicethat will be up to 100 times more accurate. Another key application willbe in vehicle guidance systems and it will eventually form the basis ofa system that will know exactly where the vehicle is on the face of theearth within a few centimeters.

An additional use will be to monitor the motion of the vehicle incomparison with that of an occupant. From this, several facts can begained. First, if the occupant moves in such a manner that is not causedby the motion of the vehicle, then the occupant must be alive.Conversely, if the driver motion is only caused by the vehicle, thenperhaps he or she is asleep or otherwise incapacitated. A given driverwill usually have a characteristic manner of operating the steeringwheel to compensate for drift on the road. If this manner changes, thenagain, the occupant may be falling asleep. If the motion of the occupantseems to be restrained relative to what a free body would do, then therewould be an indication that the seatbelt is in use, and if not, that theseatbelt is not in use or that it is too slack and needs to be retractedsomewhat.

11.5 Continuous Tracking

Previously, the output of the pattern recognition system, the neuralnetwork or combined neural network, has been the zone that the occupantis occupying. This is a somewhat difficult task for the neural networksince it calls for a discontinuous output for a continuous input. If theoccupant is in the safe seating zone, then the output may be 0, forexample and 1 if he moves into the at-risk zone. Thus, for a smallmotion there is a big change in output. On the other hand, as long asthe occupant remains in the safe seating zone, he or she can movesubstantially with no change in output. A better method is to have asthe output the position of the occupant from the airbag, for example,which is a continuous function and easier for the neural network tohandle. This also provides for a meaningful output that permits, forexample, the projection or extrapolation of the occupant's positionforward in time and thus a prediction as to when he or she will enteranother zone. This training of a neural network using a continuousposition function is an important teaching of at least one of theinventions disclosed herein.

To do continuous tracking, however, the neural network must be trainedon data that states the occupant location rather than the zone that heor she is occupying. This requires that this data be measured by adifferent system than is being used to monitor the occupant. Variouselectromagnetic systems have been tried but they tend to get foiled bythe presence of metal in the interior passenger compartment. Ultrasonicsystems have provided such information as have various optical systems.Tracking with a stereo camera arrangement using black light forillumination, for example is one technique. The occupant can even beilluminated with a UV point of light to make displacement easier tomeasure.

In addition, when multiple cameras are used in the final system, aseparate tracking system may not be required. The normalization processconducted above, for example, created a displacement value for each ofthe CCD or CMOS arrays in the assemblies 49, 50, 51, 52, and 54, (FIG.8A) or a subset thereof, which can now be used in reverse to find theprecise location of the driver's head or chest, for example, relative tothe known location of the airbag. From the vehicle geometry, and thehead and chest location information, a choice can now be made as towhether to track the head or chest for dynamic out-of-position analysis.

Tracking of the motion of the occupant's head or chest can be done usinga variety of techniques. One preferred technique is to use differentialmotion, that is, by subtracting the current image from the previousimage to determine which pixels have changed in value and by looking atthe leading edge of the changed pixels and the width of the changedpixel field, a measurement of the movement of the pixels of interest,and thus the driver, can be readily accomplished. More generally, imagesubtraction can be used to determine motion of the object represented bythe edges or more generally, the presence of an object capable ofmovement. This may be useful to discriminate between movable objects andimmovable objects. Moreover, image subtraction is not limited totracking motion of an occupant's head or chest and can be used to trackmotion of the entire occupant or other parts of the occupant.

Alternately, in another technique, a correlation function can be derivedwhich correlates the pixels in the known initial position of the head,for example, with pixels that were derived from the latest image. Thedisplacement of the center of the correlation pixels would represent themotion of the head of the occupant. A wide variety of other techniqueswill now be obvious to those skilled in the art.

In a method disclosed above for tracking motion of a vehicularoccupant's head or chest in accordance with the inventions,electromagnetic waves are transmitted toward the occupant from at leastone location, a first image of the interior of the passenger compartmentis obtained from each location, the first image being represented by amatrix of pixels, and electromagnetic waves are transmitted toward theoccupant from the same location(s) at a subsequent time and anadditional image of the interior of the passenger compartment isobtained from each location, the additional image being represented by amatrix of pixels. The additional image is subtracted from the firstimage to determine which pixels have changed in value. A leading edge ofthe changed pixels and a width of a field of the changed pixels isdetermined to thereby determine movement of the occupant from the timebetween which the first and additional images were taken. The firstimage is replaced by the additional image and the steps of obtaining anadditional image and subtracting the additional image from the firstimage are repeated such that progressive motion of the occupant isattained.

Other methods of continuous tracking include placing an ultrasonictransducer in the seatback and also on the airbag, each providing ameasure of the displacement of the occupant. Knowledge of vehiclegeometry is required here, such as the position of the seat. Thethickness of the occupant can then be calculated and two measures ofposition are available. Other ranging systems such as optical rangemeters and stereo or distance by focusing cameras could be used in placeof the ultrasonic sensors. Another system involves the placement on theoccupant of a resonator or reflector such as a radar reflector,resonating antenna, or an RFID or SAW tag. In several of these cases,two receivers and triangulation based on the time of arrival of thereturned pulses may be required.

Tracking can also be done during data collection using the same or adifferent system comprising structured light. If a separate trackingsystem is used, the structured light can be projected onto the object attime intervals in-between the taking of data with the main system. Inthis manner, the tracking system would not interfere with the imagebeing recorded by the primary system. All of the methods of obtainingthree-dimensional information described above can be implemented in aseparate tracking system.

11.6 Preprocessing

Another important feature of a system, developed in accordance with theteachings of at least one of the inventions disclosed herein, is therealization that motion of the vehicle can be used in a novel manner tosubstantially increase the accuracy of the system. Ultrasonic wavesreflect on most objects as light off a mirror. This is due to therelatively long wavelength of ultrasound as compared with light. As aresult, certain reflections can overwhelm the receiver and reduce theavailable information. When readings are taken while the occupant and/orthe vehicle is in motion, and these readings averaged over severaltransmission/reception cycles, the motion of the occupant and vehiclecauses various surfaces to change their angular orientation slightly butenough to change the reflective pattern and reduce this mirror effect.The net effect is that the average of several cycles gives a muchclearer image of the reflecting object than is obtainable from a singlecycle. This then provides a better image to the neural network andsignificantly improves the identification accuracy of the system. Thechoice of the number of cycles to be averaged depends on the systemrequirements. For example, if dynamic out-of-position is required, theneach vector must be used alone and averaging in the simple sense cannotbe used. This will be discussed more detail below. Similar techniquescan be used for other transducer technologies. Averaging, for example,can be used to minimize the effects of flickering light in camera-basedsystems.

Only rarely is unprocessed or raw data that is received from the A-to-Dconverters fed directly into the pattern recognition system. Instead, itis preprocessed to extract features, normalize, eliminate bad data,remove noise and elements that have no informational value etc.

For example, for military target recognition is common to use theFourier transform of the data rather than the data itself. This can beespecially valuable for categorization as opposed to location of theoccupant and the vehicle. When used with a modular network, for example,the Fourier transform of the data may be used for the categorizationneural network and the non-transformed data used for the positiondetermination neural network. Recently, wavelet transforms have alsobeen considered as a preprocessor.

Above, under the subject of dynamic out-of-position, it was discussedthat the position of the occupant can be used as a preprocessing filterto determine the quality of the data in a particular vector. Thistechnique can also be used in general as a method to improve the qualityof a vector of data based on the previous positions of the occupant.This technique can also be expanded to help differentiate live objectsin the vehicle from inanimate objects. For example, a forward facinghuman will change his position frequently during the travel of thevehicle whereas a box will tend to show considerably less motion. Thisis also useful, for example, in differentiating a small human from anempty seat. The motion of a seat containing a small human will besignificantly different from that of an empty seat even though theparticular vector may not show significant differences. That is, avector formed from the differences from two successive vectors isindicative of motion and thus of a live occupant.

Preprocessing can also be used to prune input data points. If eachreceiving array of assemblies, 49, 50, 51, and 54 for example (FIG. 8A),contains a matrix of 100 by 100 pixels (forming an image), then 40,000(4×100×100) pixels or data elements of information will be created eachtime the system interrogates the driver seat, for example. There aremany pixels of each image that can be eliminated as containing no usefulinformation. This typically includes the corner pixels, back of the seatand other areas where an occupant cannot reside. This pixel pruning cantypically reduce the number of pixels by up to 50 percent resulting inapproximately 20,000 remaining pixels. The output from each array isthen compared with a series of stored arrays representing differentunoccupied positions of the seat, seatback, steering wheel etc. For eacharray, each of the stored arrays is subtracted from the acquired arrayand the results analyzed to determine which subtraction resulted in thebest match. The best match is determined by such things as the totalnumber of pixels reduced below the threshold level, or the minimumnumber of remaining detached pixels, etc. Once this operation iscompleted for all four images, the position of the movable elementswithin the passenger compartment has been determined. This includes thesteering wheel angle, telescoping position, seatback angle, headrestposition, and seat position. This information can be used elsewhere byother vehicle systems to eliminate sensors that are currently being usedto sense such positions of these components. Alternately, the sensorsthat are currently on the vehicle for sensing these component positionscan be used to simplify processes described above. Each receiving arraymay also be a 256×256 CMOS pixel array as described in the paper by C.Sodini et al. referenced above greatly increasing the need for anefficient pruning process.

An alternate technique of differentiating between the occupant and thevehicle is to use motion. If the images of the passenger seat arecompared over time, reflections from fixed objects will remain staticwhereas reflections from vehicle occupants will move. This movement canbe used to differentiate the occupant from the background.

Following one or more of the subtraction processes described above, eachimage now consists of typically as many as 50 percent fewer pixelsleaving a total of approximately 10,000 pixels remaining, for the 4array 100×100 pixel case. The resolution of the images in each array cannow be reduced by combining adjacent pixels and averaging the pixelvalues. This results in a reduction to a total pixel count ofapproximately 1000. The matrices of information that contains the pixelvalues is now normalized to place the information in a location in thematrix which is independent of the seat position. The resultingnormalized matrix of 1000 pixel values can now be used as input into anartificial neural network and represents the occupancy of the seatindependent of the position of the occupant. This is a brut force methodand better methods based on edge detection and feature extraction cangreatly simplify this process as discussed below.

There are many mathematical techniques that can be applied to simplifythe above process. One technique used in military pattern recognition,as mentioned above, uses the Fourier transform of particular areas in animage to match with known Fourier transforms of known images. In thismanner, the identification and location can be determinedsimultaneously. There is even a technique used for target identificationwhereby the Fourier transforms are compared optically as mentionedherein. Other techniques utilize thresholding to limit the pixels thatwill be analyzed by any of these processes. Other techniques search forparticular features and extract those features and concentrate merely onthe location of certain of these features. (See for example the Kage etal. artificial retina publication referenced above.)

Generally, however as mentioned, the pixel values are not directly fedinto a pattern recognition system but rather the image is preprocessedthrough a variety of feature extraction techniques such as an edgedetection algorithm. Once the edges are determined, a vector is createdcontaining the location of the edges and their orientation and thatvector is fed into the neural network, for example, which performs thepattern recognition.

Another preprocessing technique that improves accuracy is to remove thefixed parts of the image, such as the seatback, leaving only theoccupying object. This can be done many ways such as by subtracting oneimage from another after the occupant has moved, as discussed above.Another is to eliminate pixels related to fixed parts of the imagethrough knowledge of what pixels to removed based on seat position andprevious empty seat analysis. Other techniques are also possible. Oncethe occupant has been isolated, then those pixels remaining can beplaced in a particular position in the neural network vector. This isakin to the fact that a human, for example, will always move his or hereyes so as to place the object under observation into the center of thefield of view, which is a small percent of the total field of view. Inthis manner the same limited number in pixels always observe the imageof the occupying item thereby removing a significant variable andgreatly improving system accuracy. The position of the occupant than canbe determined by the displacement required to put the image into theappropriate part of the vector.

The above discussion has focuses on use of four arrays, each obtainingan image of the same vehicular compartment. A different number of arrayscan be used in the invention, including only a single array, i.e., asingle camera.

11.7 Post Processing

Once the pattern recognition system has been applied to the preprocesseddata, one or more decisions are available as output. The output from thepattern recognition system is usually based on a snapshot of the outputof the various transducers unless a combination neural network withfeedback was used. Thus, it represents one epoch or time period. Theaccuracy of such a decision can usually be substantially improved ifprevious decisions from the pattern recognition system are alsoconsidered. In the simplest form, which is typically used for theoccupancy identification stage, the results of many decisions areaveraged together and the resulting averaged decision is chosen as thecorrect decision. Once again, however, the situation is quite differentfor dynamic out-of-position occupants. The position of the occupant mustbe known at that particular epoch and cannot be averaged with hisprevious position. On the other hand, there is information in theprevious positions that can be used to improve the accuracy of thecurrent decision. For example, if the new decision says that theoccupant has moved six inches since the previous decision, and, fromphysics, it is known that this could not possibly take place, then abetter estimate of the current occupant position can be made byextrapolating from earlier positions. Alternately, an occupancy positionversus time curve can be fitted using a variety of techniques such asthe least squares regression method, to the data from previous 10epochs, for example. This same type of analysis could also be applied tothe vector itself rather than to the final decision thereby correctingthe data prior to entry into the pattern recognition system. Analternate method is to train a module of a modular neural network topredict the position of the occupant based on feedback from previousresults of the module.

Summarizing, when an occupant is sitting in the vehicle during normalvehicle operation, the determination of the occupancy state can besubstantially improved by using successive observations over a period oftime. This can either be accomplished by averaging the data prior toinsertion into a neural network, or alternately the decision of theneural network can be averaged. This is known as the categorizationphase of the process. During categorization, the occupancy state of thevehicle is determined. Is the vehicle occupied by the forward facinghuman, an empty seat, a rear facing child seat, or an out-of-positionhuman? Typically many seconds of data can be accumulated to make thecategorization decision. For non-automotive vehicles this categorizationprocess may be the only process that is required. Is the containeroccupied or is it empty? If occupied is there a human or other life formpresent? Is there a hazardous chemical or a source of radioactivitypresent etc.?

When a driver senses an impending crash, he or she will typically slamon the brakes to try to slow vehicle prior to impact. If an occupant,particularly the passenger, is unbelted, he or she will begin movingtoward the airbag during this panic braking. For the purposes ofdetermining the position of the occupant, there is not sufficient timeto average data as in the case of categorization. One method is todetermine the location of the occupant using the neural network based onprevious training. The motion of the occupant can then be compared to amaximum likelihood position based on the position estimate of theoccupant at previous vectors. Thus, for example, perhaps the existenceof thermal gradients in the vehicle caused an error in the currentvector leading to a calculation that the occupant has moved 12 inchessince the previous vector. Since this could be a physically impossiblemove during ten milliseconds, the measured position of the occupant canbe corrected based on his previous positions and known velocity. If anaccelerometer is present in the vehicle and if the acceleration data isavailable for this calculation, a much higher accuracy prediction can bemade. Thus, there is information in the data in previous vectors as wellas in the positions of the occupant determined from the latest data thatcan be used to correct erroneous data in the current vector and,therefore, in a manner not too dissimilar from the averaging method forcategorization, the position accuracy of the occupant can be known withhigher accuracy.

Post processing can use a comparison of the results at each timeinterval along with a test of reasonableness to remove erroneousresults. Also averaging through a variety of techniques can improve thestability of the output results. Thus the output of a combination neuralnetwork is not necessarily the final decision of the system.

One principal used in a preferred implementation of at least oneinvention herein is to use images of different views of the occupant tocorrelate with known images that were used to train a neural network forvehicle occupancy. Then carefully measured positions of the known imagesare used to locate particular parts of the occupant such as his or herhead, chest, eyes, ears, mouth, etc. An alternate approach is to make athree-dimensional map of the occupant and to precisely locate thesefeatures using neural networks, sensor fusion, fuzzy logic or otherpattern recognition techniques. One method of obtaining athree-dimensional map is to utilize a scanning laser radar system wherethe laser is operated in a pulse mode and the distance from the objectbeing illuminated is determined using range gating in a manner similarto that described in various patents on micropower impulse radar toMcEwan. (See, for example, U.S. Pat. Nos. 5,457,394 and 5,521,600) Manyother methods of obtaining a 3D representation can be used as discussedin detail above. This post processing step allows the determination ofoccupant parts from the image once the object is classified as anoccupant.

Many other post processing techniques are available as discussed herein.

11.8 An Example of Image Processing

As an example of the above concepts, a description of a single imageroptical occupant classification system will now be presented.

11.8.1 Image Preprocessing

A number of image preprocessing filters have been implemented, includingnoise reduction, contrast enhancement, edge detection, image downsampling and cropping, etc. and some of them will now be discussed.

The Gaussian filter, for example, is very effective in reducing noise inan image. The Laplacian filter can be used to detect edges in an image.The result from a Laplacian filter plus the original image produces anedge-enhanced image. Both the Gaussian filter and the Laplacian filtercan be implemented efficiently when the image is scanned twice. Theoriginal Kirsch filter consists of 8 filters that detect edges of 8different orientations. The max Kirsch filter, however, uses a singlefilter that detects (but does not distinguish) edges of all 8 differentorientations.

The histogram-based contrast enhancement filter improves image contrastby stretching pixel grayscale values until a desired percentage ofpixels are suppressed and/or saturated. The wavelet-based enhancementfilter modifies an image by performing multilevel wavelet decompositionand then applies a nonlinear transfer function to the detailcoefficients. This filter reduces noise if the nonlinear transferfunction suppresses the detail coefficients, and enhances the image ifthe nonlinear transfer function retains and increases the significantdetail coefficients. A total of 54 wavelet functions from 7 families,for example, have been implemented.

Mathematical morphology has been proven to be a powerful tool for imageprocessing (especially texture analysis). For example, the grayscalemorphological filter that has been implemented by the current assigneeincludes the following operators: dilation, erosion, close, open, whitetop hat, black top hat, h-dome, and noise removal. The structure elementis totally customizable. The implementation uses fast algorithms such asvan Herk/Gil-Werman's dilation/erosion algorithm, and Luc Vincent'sgrayscale reconstruction algorithm.

Sometimes using binary images instead of grayscale images increases thesystem robustness. The binarization filter provides 3 different ways toconvert a grayscale image into a binary image: 1) using a constantthreshold; 2) specifying a white pixel percentage; 3) Otsu's minimumdeviation method. The image down-size filter performs imagedown-sampling and image cropping. This filter is useful for removingunwanted background (but limited to preserving a rectangular region).Image down-sampling is also useful because our experiments show that,given the current accuracy requirement, using a lower resolution imagefor occupant position detection does not degrade the system performance,and is more computationally efficient.

Three other filters that were implemented provide maximum flexibility,but require more processing time. The generic in-frame filter implementsalmost all known and to be developed window-based image filters. Itallows the user to specify a rectangular spatial window, and define amathematical function of all the pixels within the window. This coversalmost all well-known filters such as averaging, median, Gaussian,Laplacian, Prewit, Sobel, and Kirsch filters. The generic cross-framefilter implements almost all known and to be developed time-basedfilters for video streams. It allows the user to specify a temporalwindow, and define a mathematical function of all the frames within thewindow. The pixel transfer filter provides a flexible way to transforman image. A pixel value in the resulting image is a customizablefunction of the pixel coordinates and the original pixel value. Thepixel transfer filter is useful in removing unwanted regions withirregular shapes.

FIG. 59 of the '996 application shows some examples of the preprocessingfilters that have been implemented. FIG. 59(1) shows the original image.FIG. 59(2) shows the result from a histogram-based contrast enhancementfilter. FIG. 59(3) shows the fading effect generated using a pixeltransfer filter where the transfer function is defined as

$\frac{1}{14}z^{1.5}{^{- {0.0001{\lbrack{{({x - 60})}^{2} + {({y - 96})}^{2}}\rbrack}}}.}$

FIG. 59(4) shows the result from a morphological filter followed by ahistogram-based contrast enhancement filter. The h-dome operator wasused with the dome height=128. One can see that the h-dome operatorpreserves bright regions and regions that contain significant changes,and suppresses dark and flat regions. FIG. 59(5) shows the edgesdetected using a Laplacian filter. FIG. 59(6) shows the result from aGaussian filter followed by a max Kirsch filter, a binarization filterthat uses Otsu's method, and a morphological erosion that uses a 3×3flat structure element.

11.8.2 Feature Extraction Algorithm

The image size in the current classification system is 320×240, i.e.76,800 pixels, which is too large for the neural network to handle. Inorder to reduce the amount of the data while retaining most of theimportant information, a good feature extraction algorithm is needed.One of the algorithms that was developed includes three steps:

1) Divide the whole image into small rectangular blocks.

2) Calculate a few feature values from each block.

3) Line up the feature values calculated from individual blocks and thenapply normalization.

By dividing the image into blocks, the amount of the data is effectivelyreduced while most of the spatial information is preserved.

This algorithm was derived from a well-known algorithm that has beenused in applications such as handwriting recognition. For most of thedocument related applications, binary images are usually used. Studieshave shown that the numbers of the edges of different orientations in ablock are very effective feature values for handwriting recognition. Forour application where grayscale images are used, the count of the edgescan be replaced by the sum of the edge strengths that are defined as thelargest differences between the neighboring pixels. The orientation ofan edge is determined by the neighboring pixel that produces the largestdifference between itself and the pixel of interest (see FIG. 60 of the'996 application).

FIGS. 61 and 62 of the '996 application show the edges of eightdifferent orientations that are detected using Kirsch filters. Thefeature values that are calculated from these edges are also shown.Besides Kirsch filters, other edge detection methods such as Prewit andSobel filters were also implemented.

Besides the edges, other information can also be used as the featurevalues. FIG. 63 of the '996 application shows the feature valuescalculated from the block-average intensities and deviations. Ourstudies show that the deviation feature is less effective than the edgeand the intensity features.

The edge detection techniques are usually very effective for findingsharp (or abrupt) edges. But for blunt (or rounded) edges, most of thetechniques are not effective at all. These kinds of edges also containuseful information for classification. In order to utilize suchinformation, a multi-scale feature extraction technique was developed.In other words, after the feature extraction algorithm was applied tothe image of the original size, a 50% down-sampling was done and thesame feature extraction algorithm (with the same block size) was appliedto the image of reduced size. If it is desired to find even blunteredges, this technique can be applied again to the down-sampled image.

11.8.3 Modular Neural Network Architecture

The camera-based optical occupant classification system described herewas designed to be a stand-alone system whose only input is the imagefrom the camera. Once an image is converted into a feature vector, theclassification decision can be made using any pattern recognitiontechnique. A vast amount of evidence in literature shows that a neuralnetwork technique is particularly effective in image-based patternrecognition applications.

In this application, the patterns of the feature vectors are extremelycomplex. Therefore, with reference to FIGS. 104-107 of the '881application, it has been found that a modular approach is extremelyeffective with such complex systems.

11.8.4 Post Neural Network Processing

As discussed in the '881 application with reference to FIGS. 108-111,post-processing filters can be used to eliminate the random fluctuationsin the neural network output.

11.8.5 Data Collection and Neural Network Training

Schemes to collect data during the night and during the day arediscussed in the '881 application with reference to FIGS. 112-117

11.8.6 Conclusions and Discussions

A symmetrical neural network architecture, shown in FIG. 107 of the '881application, was developed after the system reported here. Results provethat this architecture gives better performance than the otherarchitectures. With this architecture, it is possible to reducemisclassifications by replacing the weak classifications with“undetermined” states. More importantly, this architecture provides away to identify “unseen” patterns.

Development of an optical occupant sensing system requires many softwaretools whose functionalities include: communication with hardware,assisting data collection, analyzing and converting data, trainingmodular neural networks, evaluating and demonstrating systemperformance, and evaluating new algorithms. Major software componentsare shown in FIG. 118 of the '881 application.

It is important to note that the classification accuracies reported hereare based on single images and when the post processing steps areincluded the overall system accuracy approaches 100%. This is asubstantial improvement over previous systems even thought it is basedon a single camera. Although this system is capable of dynamic tracking,some additional improvement can be obtained through the addition of asecond camera. Nevertheless, the system as described herein is costcompetitive with a weight only system and substantially more accurate.This system is now ready for commercialization where the prototypesystem described herein is made ready for high volume serial production.

12. Optical Correlators

A great deal of effort has been ongoing to develop fast optical patternrecognition systems to allow military vehicles such as helicopters tolocate all of the enemy vehicles in a field of view. Some of the systemsthat have been developed are called optical correlation systems and havethe property that the identification and categorization of variousobjects in the field of view happens very rapidly. A helicopter, forexample coming onto a scene with multiple tanks and personnel carriersin a wide variety of poses and somewhat camouflaged can locate, identifyand count all such vehicles in a fraction of a second. The cost of thesesystems has been prohibitively expensive for their use in automobilesfor occupant tracking or for collision avoidance but this is changing.

Theoretically system performance is simple. The advantage of opticalcorrelation approach is that correlation function is calculated almostinstantly, much faster that with microprocessors and neural networks,for example. In simplest case one looks for correlation of an inputimage with reference samples. The sample which has the largestcorrelation peak is assumed as a match. In practice, the system is basedon a training set of reference samples. Special filters are constructedfor correlation with input image. Filters are used in order to reducenumber of correlations to calculate. The output of the filters, theresult of the correlation, is frequently a set of features. Finally thefeatures are fed into a classifier for decision making. This classifiercan use Neural Networks.

The main bottleneck of optical correlators is large number of filters,or reference image samples, that are required. For example, if it isrequirement to detect 10 different types of objects at differentorientation, scale and illumination conditions, every modificationfactor enlarges number of filters for feature selection or correlationby factor of approximately 10. So, in a real system one may have toinput 10,000 filters or reference images. Most correlators are able tofind correlation of an input image with about of 5-20 filters duringsingle correlation cycle. In other words the reference image contains5-20 filters. Therefore during decision making cycle one needs to feedinto correlator and find correlation with approximately 1000 filters.

If the problem is broken down, as was done with modular neural networks,then the classification stage may take on the order of a second whilethe tracking stage can be done perhaps in a millisecond.

U.S. Pat. Nos. 5,473,466 and 5,051,738 describe a miniature highresolution display system for use with heads up displays forinstallation into the helmets of fighter pilots. This system, which isbased on a thin garnet crystal, requires very little power and maintainsa particular display until display is changed. Thus, for example, ifthere is a loss of power the display will retain the image that was lastdisplayed. This technology has the capability of producing a very smallheads up display unit as will be described more detail below. Thistechnology has also been used as a spatial light monitor for patternrecognition based on optical correlation. Although this technology hasbeen applied to military helicopters, it has previously not been usedfor occupant sensing, collision avoidance, anticipatory sensing, blindspot monitoring or any other ground vehicle application.

Although the invention described herein is not limited to a particularspatial light monitor (SLM) technology, the preferred or best modetechnology is to use the garnet crystal system described in U.S. Pat.No. 5,473,466. Although the system has never been applied toautomobiles, it has significant advantages over other systemsparticularly in the resolution and optical intensity areas. Theresolution of the garnet crystals as manufactured by Revtek isapproximately 600 by 600 pixels. The size of the crystal is typically 1cm square.

Basically, the optical correlation pattern recognition system works asfollows. Stored in a computer are many Fourier transforms of images ofobjects that the system should identify. For collision avoidance, theseinclude cars, trucks, deer or other animals, pedestrians, motorcycles,bicycles, or any other objects that could occur on a roadway. For aninterior monitoring, these objects could include faces (particularlyones that are authorized to operate the vehicle), eyes, ears, childseats, children, adults of all sizes etc. The image from the scene thatis captured by the lens is fed through a diffraction grating thatoptically creates the Fourier transform of the scene and projects itthrough SLM such as the garnet crystal of the '466 patent. The SLM issimultaneously fed and displays the Fourier stored transforms and acamera looks at the light that comes through the SLM. If there is amatch then the camera sees a spike that locates the matching objects inthe scene, there can be many such objects, all are found. The mainadvantage of this system over neural network pattern recognition systemsis speed since it is all done optically and in parallel.

For collision avoidance, for example, many vehicles can be easilyclassified and tracked. For occupant sensing, the occupant's eyes can betracked even if he is rapidly moving his head and the occupant herselfcan be tracked during a crash.

13. Other Products, Outputs, Features

Once the occupancy state of the seat (or seats) in the vehicle or of thevehicle itself, as in a cargo container, truck trailer or railroad car,is known, this information can be used to control or affect theoperation of a significant number of vehicular systems, components anddevices. That is, the systems, components and devices in the vehicle canbe controlled and perhaps their operation optimized in consideration ofthe occupancy of the seat(s) in the vehicle or of the vehicle itself.Thus, the vehicle includes a control system coupled to the processor forcontrolling a component or device in the vehicle in consideration of theoutput indicative of the current occupancy state of the seat obtainedfrom the processor. The component or device can be an airbag systemincluding at least one deployable airbag whereby the deployment of theairbag is suppressed, for example, if the seat is occupied by arear-facing child seat, or otherwise the parameters of the deploymentare controlled. Thus, the seated-state detecting unit described abovemay be used in a component adjustment system and method described belowwhen the presence of a human being occupying the seat is detected. Thecomponent can also be a telematics system such as the Skybitz or OnStarsystems where information about the occupancy state of the vehicle, orchanges in that state, can be sent to a remote site.

The component adjustment system and methods in accordance with theinvention can automatically and passively adjust the component based onthe morphology of the occupant of the seat. As noted above, theadjustment system may include the seated-state detecting unit describedabove so that it will be activated if the seated-state detecting unitdetects that an adult or child occupant is seated on the seat, that is,the adjustment system will not operate if the seat is occupied by achild seat, pet or inanimate objects. Obviously, the same system can beused for any seat in the vehicle including the driver seat and thepassenger seat(s). This adjustment system may incorporate the samecomponents as the seated-state detecting unit described above, that is,the same components may constitute a part of both the seated-statedetecting unit and the adjustment system, for example, the weightmeasuring system.

The adjustment system described herein, although improved over the priorart, will at best be approximate since two people, even if they areidentical in all other respects, may have a different preferred drivingposition or other preferred adjusted component location or orientation.A system that automatically adjusts the component, therefore, shouldlearn from its errors. Thus, when a new occupant sits in the vehicle,for example, the system automatically estimates the best location of thecomponent for that occupant and moves the component to that location,assuming it is not already at the best location. If the occupant changesthe location, the system should remember that change and incorporate itinto the adjustment the next time that person enters the vehicle and isseated in the same seat. Therefore, the system need not make a perfectselection the first time but it should remember the person and theposition the component was in for that person. The system, therefore,makes one, two or three measurements of morphological characteristics ofthe occupant and then adjusts the component based on an algorithm. Theoccupant will correct the adjustment and the next time that the systemmeasures the same measurements for those measurement characteristics, itwill set the component to the corrected position. As such, preferredcomponents for which the system in accordance with the invention is mostuseful are those which affect a driver of the vehicle and relate to thesensory abilities of the driver, i.e., the mirrors, the seat, thesteering wheel and steering column and accelerator, clutch and brakepedals.

Thus, although the above description mentions that the airbag system canbe controlled by the control circuitry 20 (FIG. 1), any vehicularsystem, component or subsystem can be controlled based on theinformation or data obtained by transmitter and/or receiver assemblies6, 8, 9 and 10. Control circuitry 20 can be programmed or trained, iffor example a neural network is used, to control heating anair-conditioning systems based on the presence of occupants in certainpositions so as to optimize the climate control in the vehicle. Theentertainment system can also be controlled to provide sound only tolocations at which occupants are situated. There is no limit to thenumber and type of vehicular systems, components and subsystems that canbe controlled using the analysis techniques described herein.

Furthermore, if multiple vehicular systems are to be controlled bycontrol circuitry 20, then these systems can be controlled by thecontrol circuitry 20 based on the status of particular components of thevehicle. For example, an indication of whether a key is in the ignitioncan be used to direct the control circuitry 20 to either control anairbag system (when the key is present in the ignition) or an antitheftsystem (when the key is not present in the ignition). Control circuitry20 would thus be responsive to the status of the ignition of the motorvehicle to perform one of a plurality of different functions. Moreparticularly, the pattern recognition algorithm, such as the neuralnetwork described herein, could itself be designed to perform in adifferent way depending on the status of a vehicular component such asthe detected presence of a key in the ignition. It could provide oneoutput to control an antitheft system when a key is not present andanother output when a key is present using the same inputs from thetransmitter and/or receiver assemblies 6, 8, 9 and 10.

The algorithm in control circuitry 20 can also be designed to determinethe location of the occupant's eyes either directly or indirectlythrough a determination of the location of the occupant and anestimation of the position of the eyes therefrom. As such, the positionof the rear view mirror 55 can be adjusted to optimize the driver's usethereof.

Once a characteristic of the object is obtained, it can be used fornumerous purposes. For example, the processor can be programmed tocontrol a reactive component, system or subsystem 103 in FIG. 17 basedon the determined characteristic of the object. When the reactivecomponent is an airbag assembly including one or more airbags, theprocessor can control one or more deployment parameters of theairbag(s).

The apparatus can operate in a manner as illustrated in FIG. 7 whereinas a first step 335, one or more images of the environment are obtained.One or more characteristics of objects in the images are determined at336, using, for example, pattern recognition techniques, and then one ormore components are controlled at 337 based on the determinedcharacteristics. The process of obtaining and processing the images, orthe processing of data derived from the images or data representative ofthe images, is periodically continued at least throughout the operationof the vehicle.

13.1 Control of Passive Restraints

Use of the vehicle interior monitoring system to control the deploymentof an airbag is discussed in U.S. Pat. No. 5,653,462. In that case, thecontrol is based on the use of a pattern recognition system, such as aneural network, to differentiate between the occupant and hisextremities in order to provide an accurate determination of theposition of the occupant relative to the airbag. If the occupant issufficiently close to the airbag module that he is more likely to beinjured by the deployment itself than by the accident, the deployment ofthe airbag is suppressed. This process is carried further by theinterior monitoring system described herein in that the nature oridentity of the object occupying the vehicle seat is used to contributeto the airbag deployment decision. FIG. 4 shows a side view illustratingschematically the interface between the vehicle interior monitoringsystem of at least one of the inventions disclosed herein and thevehicle airbag system 44. A similar system can be provided for thepassenger as described in U.S. Pat. No. 6,820,897.

In this embodiment, ultrasonic transducers 8 and 9 transmit bursts ofultrasonic waves that travel to the occupant where they are reflectedback to transducers or receptors/receivers 8 and 9. The time periodrequired for the waves to travel from the generator and return is usedto determine the distance from the occupant to the airbag as describedin U.S. Pat. No. 5,653,462, i.e., and thus may also be used to determinethe position or location of the occupant. An optical imager based systemwould also be appropriate. In the invention, however, the portion of thereturn signal that represents the occupants' head or chest, has beendetermined based on pattern recognition techniques such as a neuralnetwork. The relative velocity of the occupant toward the airbag canthen be determined, by Doppler principles or from successive positionmeasurements, which permits a sufficiently accurate prediction of thetime when the occupant would become proximate to the airbag. Bycomparing the occupant relative velocity to the integral of the crashdeceleration pulse, a determination as to whether the occupant is beingrestrained by a seatbelt can also be made which then can affect theairbag deployment initiation decision. Alternately, the mere knowledgethat the occupant has moved a distance that would not be possible if hewere wearing a seatbelt gives information that he is not wearing one.

Another method of providing a significant improvement to the problem ofdetermining the position of the occupant during vehicle deceleration isto input the vehicle deceleration directly into the occupant sensingsystem. This can be done through the use of the airbag crash sensoraccelerometer or a dedicated accelerometer can be used. Thisdeceleration or its integral can be entered directly into the neuralnetwork or can be integrated through an additional post-processingalgorithm. Post processing in general is discussed in section 11.7. Onesignificant advantage of neural networks is their ability to efficientlyuse information from any source. It is the ultimate “sensor fusion”system.

A more detailed discussion of this process and of the advantages of thevarious technologies, such as acoustic or electromagnetic, can be foundin SAE paper 940527, “Vehicle Occupant Position Sensing” by Breed etal., In this paper, it is demonstrated that the time delay required foracoustic waves to travel to the occupant and return does not prevent theuse of acoustics for position measurement of occupants during the crashevent. For position measurement and for many pattern recognitionapplications, ultrasonics is the preferred technology due to the lack ofadverse health effects and the low cost of ultrasonic systems comparedwith either camera, laser or radar based systems. This situation haschanged, however, as the cost of imagers has come down. The mainlimiting feature of ultrasonics is the wavelength, which places alimitation on the size of features that can be discerned. Opticalsystems, for example, are required when the identification of particularindividuals is desired.

In the embodiment shown in FIG. 8A, transmitter/receiver assemblies 49,50, 51 and 54 emit infrared waves that reflect off of the head and chestof the driver and return thereto. Periodically, the device, as commandedby control circuitry 20, transmits a pulse of infrared waves and thereflected signal is detected by the same (i.e. the LEDs and imager arein the same housing) or a different device. The transmitters can eithertransmit simultaneously or sequentially. An associated electroniccircuit and algorithm in control circuitry 20 processes the returnedsignals as discussed above and determines the location of the occupantin the passenger compartment. This information is then sent to the crashsensor and diagnostic circuitry, which may also be resident in controlcircuitry 20 (programmed within a control module), which determines ifthe occupant is close enough to the airbag that a deployment might, byitself, cause injury which exceeds that which might be caused by theaccident itself. In such a case, the circuit disables the airbag systemand thereby prevents its deployment.

In an alternate case, the sensor algorithm assesses the probability thata crash requiring an airbag is in process and waits until thatprobability exceeds an amount that is dependent on the position of theoccupant. Thus, for example, the sensor might decide to deploy theairbag based on a need probability assessment of 50%, if the decisionmust be made immediately for an occupant approaching the airbag, butmight wait until the probability rises above 95% for a more distantoccupant. In the alternative, the crash sensor and diagnostic circuitryoptionally resident in control circuitry 20 may tailor the parameters ofthe deployment (time to initiation of deployment, rate of inflation,rate of deflation, deployment time, etc.) based on the current positionand possibly velocity of the occupant, for example a depowereddeployment.

In another implementation, the sensor algorithm may determine the ratethat gas is generated to affect the rate that the airbag is inflated.One method of controlling the gas generation rate is to control thepressure in the inflator combustion chamber. The higher the internalpressure the faster gas is generated. Once a method of controlling thegas combustion pressure is implemented, the capability exists tosignificantly reduce the variation in inflator properties withtemperature. At lower temperatures the pressure control system wouldincrease the pressure in the combustion chamber and at higher ambienttemperatures it would reduce the pressure. In all of these cases, theposition of the occupant can be used to affect the deployment of theairbag as to whether or not it should be deployed at all, the time ofdeployment and/or the rate of inflation.

The applications described herein have been illustrated using the driverand sometimes the passenger of the vehicle. The same systems ofdetermining the position of the occupant relative to the airbag apply toa driver, front and rear seated passengers, sometimes requiring minormodifications. It is likely that the sensor required triggering timebased on the position of the occupant will be different for the driverthan for the passenger. Current systems are based primarily on thedriver with the result that the probability of injury to the passengeris necessarily increased either by deploying the airbag too late or byfailing to deploy the airbag when the position of the driver would notwarrant it but the passenger's position would. With the use of occupantposition sensors for the passenger and driver, the airbag system can beindividually optimized for each occupant and result in furthersignificant injury reduction. In particular, either the driver orpassenger system can be disabled if either the driver or passenger isout-of-position or if the passenger seat is unoccupied.

There is almost always a driver present in vehicles that are involved inaccidents where an airbag is needed. Only about 30% of these vehicles,however, have a passenger. If the passenger is not present, there isusually no need to deploy the passenger side airbag. The occupantmonitoring system, when used for the passenger side with proper patternrecognition circuitry, can also ascertain whether or not the seat isoccupied, and if not, can disable the deployment of the passenger sideairbag and thereby save the cost of its replacement. The same strategyapplies also for monitoring the rear seat of the vehicle. Also, atrainable pattern recognition system, as used herein, can distinguishbetween an occupant and a bag of groceries, for example. Finally, therehas been much written about the out-of-position child who is standing orotherwise positioned adjacent to the airbag, perhaps due to pre-crashbraking. The occupant position sensor described herein can prevent thedeployment of the airbag in this situation as well as in the situationof a rear facing child seat as described above.

As discussed herein, occupant sensors can also be used for monitoringthe rear seats of the vehicle for the purpose, among others, ofcontrolling airbag or other restraint deployment.

13.2 Seat, Seatbelt, Steering Wheel and Pedal Adjustment and Resonators

13.3 Side Impacts

FIG. 10 is an angular perspective overhead view of a vehicle 405 aboutto be impacted in the side by an approaching vehicle 406, where vehicle405 is equipped with an anticipatory sensor system showing a transmitter408 transmitting electromagnetic, such as infrared, waves toward vehicle406. This is one example of many of the uses of the instant inventionfor exterior monitoring. The transmitter 408 is connected to anelectronic module 412. Module 412 contains circuitry 413 to drivetransmitter 408 and circuitry 414 to process the returned signals fromreceivers 409 and 410 which are also coupled to module 412. Circuitry414 contains a processor such as a neural computer 415 or microprocessorwith a pattern recognition algorithm, which performs the patternrecognition determination based on signals from receivers 409 and 410.Receivers 409 and 410 are mounted onto the B-Pillar of the vehicle andare covered with a protective transparent cover. An alternate mountinglocation is shown as 411 which is in the door window trim panel wherethe rear view mirror (not shown) is frequently attached. One additionaladvantage of this system is the ability of infrared to penetrate fog andsnow better than visible light which makes this technology particularlyapplicable for blind spot detection and anticipatory sensingapplications. Although it is well known that infrared can besignificantly attenuated by both fog and snow, it is less so than visuallight depending on the frequency chosen. (See for example L. A. Klein,Millimeter-Wave and Infrared Multisensor Design and Signal Processing,Artech House, Inc, Boston 1997, ISBN 0-89006-764-3).

13.4 Exterior Monitoring

Referring now to FIGS. 52 and 56, the same system can also be used forthe detection of objects in the blind spots and other areas surroundingthe vehicle and the image displayed for the operator to see or a warningsystem activated, if the operator attempts to change lanes, for example.In this case, the mounting location must be chosen to provide a goodview along the side of the vehicle in order to pick up vehicles whichare about to pass the subject vehicle 710. Each of the locations 408,409 and 410 provide sufficient field of view for this applicationalthough the space immediately adjacent to the vehicle could be missed.Alternate locations include mounting onto the outside rear view mirrorassembly or the addition of a unit in the rear window or C-Pillar, inwhich case, the contents of areas other than the side of the vehiclewould be monitored. Using several receivers in various locations asdisclosed above would provide for a monitoring system which monitors allof the areas around the vehicle. The mirror location, however, doesleave the device vulnerable to being covered with ice, snow and dirt.

In many cases, neural networks are used to identify objects exterior ofthe vehicle and then an icon can be displayed on a heads-up display, forexample, which provides control over the brightness of the image andpermits the driver to more easily recognize the object.

In both cases of the anticipatory sensor and blind spot detector, theinfrared transmitter and imager array system provides mainly imageinformation to permit recognition of the object in the vicinity ofvehicle 710, whether the object is alongside the vehicle, in a blindspot of the driver, in front of the vehicle or behind the vehicle, theposition of the object being detected being dependent on the positionand orientation of the receiver(s). To complete the process, distanceinformation is also require as well as velocity information, which canin general be obtained by differentiating the position data or byDoppler analysis. This can be accomplished by any one of the severalmethods discussed above, such as with a pulsed laser radar system,stereo cameras, focusing system, structured light as well as with aradar system.

Radar systems, which may not be acceptable for use in the interior ofthe vehicle, are now commonly used in sensing applications exterior tothe vehicle, police radar being one well-known example. Miniature radarsystems are now available which are inexpensive and fit within theavailable space. Such systems are disclosed in the McEwan patentsdescribed above. Another advantage of radar in this application is thatit is easy to get a transmitter with a desirable divergence angle sothat the device does not have to be aimed. One particularly advantageousmode of practicing the invention for these cases, therefore, is to useradar and a second advantageous mode is the pulsed laser radar system,along with an imager array, although the use of two such arrays or theacoustical systems are also good choices. The acoustical system has thedisadvantage of being slower than the laser radar device and must bemounted outside of the vehicle where it may be affected by theaccumulation of deposits onto the active surface. If a radar scanner isnot available it is difficult to get an image of objects approaching thevehicle so that the can be identified. Note that the ultimate solutionto monitoring of the exterior of the vehicle may lay with SWIR, MWIR andLWIR if the proper frequencies are chosen that are not heavilyattenuated by fog, snow and other atmospheric systems. The QWIP systemdiscussed above or equivalent would be a candidate if the coolingrequirement can be eliminated or the cost of cooling the imaging chipreduced. Finally, terahertz frequencies (approximately 0.1-5 THz) arebeginning to show promise for this application. They can be generatedusing laser type devices and yet have almost the fog penetration abilityof mm wave radar.

Another innovation involves the use of multiple frequencies forinterrogating the environment surrounding a vehicle and in particularthe space in front of the vehicle. Different frequencies interactdifferently with different materials. An example given by some to showthat all such systems have failure modes is the case of a box that inone case contains a refrigerator while in another case a box of the samesize that is empty. It is difficult to imagine how such boxes can resideon a roadway in front of a traveling vehicle but perhaps it fell off ofa truck. Using optics it would be difficult if not impossible to makethe distinction, however, some frequencies will penetrate a cardboardbox exposing the refrigerator. One might ask, what happens if the box ismade of metal? So there will always be rare cases where a distinctioncannot be made. Nevertheless, a calculation can be made of the cost andbenefits to be derived by fielding such a system that might occasionallymake a mistake or, better, defaults to no system when it is in doubt.

In a preferred implementation, transmitter 408 is an infraredtransmitter and receivers 409, 410 and 411 are CMOS transducers thatreceive the reflected infrared waves from vehicle 406. In theimplementation shown in FIG. 10, an exterior airbag 416 is shown whichdeploys in the event that a side impact is about to occur as describedin U.S. Pat. No. 6,343,810.

Referring now to FIG. 11, a schematic of the use of one or morereceivers 409, 410, 411 to affect another system in the vehicle isshown. The general exterior monitoring system, or blind spot monitoringsystem if the environment exterior of the vehicle is not viewable by thedriver in the normal course of driving the vehicle, includes one or morereceivers 409, 410, 411 positioned at various locations on the vehiclefor the purpose of receiving waves from the exterior environment.Instead of waves, and to the extent different than waves, the receivers409, 410, 411 could be designed to receiver energy or radiation.

The waves received by receivers 409, 410, 411 contain information aboutthe exterior objects in the environment, such waves either having beengenerated by or emanating from the exterior objects or reflected fromthe exterior objects such as is the case when the optional transmitter408 is used. The electronic module/processor 412 contains the necessarycircuitry 413,414 and a trained pattern recognition system (e.g., neuralcomputer 415) to drive the transmitter 408 when present and process thereceived waves to provide a classification, identification and/orlocation of the exterior object. The classification, identificationand/or location is then used to control a vehicle system 420, such asshowing an image on a display viewable to the driver. Also, theclassification, identification or location of the objects could be usedfor airbag control, i.e., control of the deployment of the exteriorairbag 416 (or any other airbags for that matter), for the control ofthe headlight dimmers (as discussed with reference to FIG. 12) or ingeneral, for any other system whose operation might be changed based onthe presence of exterior objects. The actual headlight dimmer mechanismmay be any controllable dimming mechanism known to those skilled in theart.

FIG. 13 shows the components for measuring the position of an object inan environment of or about the vehicle. A light source 425 directsmodulated light into the environment and at least one light-receivingpixel or an array of pixels 427 receives the modulated light afterreflection by any objects in the environment. A processor 428 determinesthe distance between any objects from which the modulated light isreflected and the light source based on the reception of the modulatedlight by the pixel(s) 427. To provide the modulated light, a device orcomponent for modulating a frequency of the light 426 are provided.Also, a device for providing a correlation pattern in a form of codedivision modulation of the light can be used. The pixel may be a photodiode such as a PIN or avalanche diode.

The processor 428 includes appropriate circuitry to determine thedistance between any objects from which any pulse of light is reflectedand the light source 425. For example, the processor 428 can determinethis distance based on a difference in time between the emission of apulse of light by the light source 425 and the reception of light by thepixel 427.

FIG. 12 illustrates the exterior monitoring system for use in detectingthe headlights of an oncoming vehicle or the taillights of a vehicle infront of vehicle 259 for the purpose of, for example, dimming theheadlights of the vehicle 259 while the oncoming vehicle is in the fieldof view of the headlights. In this embodiment, the imager array 429 isdesigned to be sensitive to visible light and a separate source ofillumination is not used. Once again for some applications, the key tothis technology is the use of trained pattern recognition algorithms andparticularly the artificial neural network. Here, as in the other casesabove and in patents and patent applications referenced above, thepattern recognition system is trained to recognize the pattern of theheadlights of an oncoming vehicle or the tail lights of a vehicle infront of vehicle 259 and to then dim the headlights when either of theseconditions is sensed. It is also trained to not dim the lights for otherreflections such as reflections off of a sign post or the roadway. Oneproblem is to differentiate taillights where dimming is desired fromdistant headlights where dimming is not desired. Three techniques areused: (i) measurement of the spacing of the light sources, (ii)determination of the location of the light sources relative to thevehicle, and (iii) use of a red filter where the brightness of the lightsource through the filter is compared with the brightness of theunfiltered light. In the case of the taillight, the brightness of thered filtered and unfiltered light is nearly the same while there is asignificant difference for the headlight case. In this situation, eithertwo CCD arrays are used, one with a filter, or a filter which can beremoved either electrically, such as with a liquid crystal, ormechanically.

The environment surrounding the vehicle can be determined using aninterior mounted camera that looks out of the vehicle. The status of thesun (day or night), the presence of rain, fog, snow, etc can thus bedetermined.

The information provided by the exterior monitoring system can becombined with the interior monitoring system in order to optimize bothsystems for the protection of the occupants.

Although several preferred embodiments are illustrated and describedabove, there are possible combinations using other signals and sensorsfor the components and different forms of the neural networkimplementation or different pattern recognition technologies thatperform the same functions which can be utilized in accordance with theinvention. Also, although the neural network and modular neural networkshave been described as an example of one means of pattern recognition,other pattern recognition means exist and still others are beingdeveloped which can be used to identify potential component failures bycomparing the operation of a component over time with patternscharacteristic of normal and abnormal component operation. In addition,with the pattern recognition system described above, the input data tothe system may be data which has been pre-processed rather than the rawsignal data either through a process called “feature extraction” or byvarious mathematical transformations. Also, any of the apparatus andmethods disclosed herein may be used for diagnosing the state ofoperation or a plurality of discrete components.

Although several preferred embodiments are illustrated and describedabove, there are possible combinations using other geometries, sensors,materials and different dimensions for the components that perform thesame functions. At least one of the inventions disclosed herein is notlimited to the above embodiments and should be determined by thefollowing claims. There are also numerous additional applications inaddition to those described above. Many changes, modifications,variations and other uses and applications of the subject inventionwill, however, become apparent to those skilled in the art afterconsidering this specification and the accompanying drawings whichdisclose the preferred embodiments thereof. All such changes,modifications, variations and other uses and applications which do notdepart from the spirit and scope of the invention are deemed to becovered by the invention which is limited only by the following claims.

1. A control system for automatically controlling headlights of avehicle, comprising: an optical system for imaging external sources oflight within a predetermined field of view; and an image processingsystem for processing images from said optical system and providing acontrol signal for controlling the headlights as a function of theprocessed images.
 2. The control system of claim 1, wherein theprocessing of the images by said image processing system includesidentifying a source of radiation in the images, the control signalbeing provided to dim the headlights when a source of radiation in theimages is identified as headlight or taillight of another vehicle. 3.The control system of claim 1, wherein said image processing systemcomprises trained pattern recognition means for processing the images toidentify the source of radiation.
 4. The control system of claim 3,wherein said pattern recognition means are structured and arranged toapply a pattern recognition algorithm generated from data of possiblesources of radiation including headlights of vehicles and patterns ofreceived radiation from the possible sources.
 5. The control system ofclaim 4, wherein the possible sources of radiation further includestaillights of vehicles.
 6. The control system of claim 3, wherein saidtrained pattern recognition means comprise a neural network.
 7. Thecontrol system of claim 1, wherein said optical system comprises a CCDarray.
 8. The control system of claim 1, wherein said optical system isarranged on a rear view mirror in an interior of the vehicle.
 9. Thecontrol system of claim 1, wherein said optical system is arranged on apart of the vehicle that is not movable relative to a frame of thevehicle.
 10. The control system of claim 1, wherein said optical systemincludes an image array sensor containing a plurality of pixels.
 11. Thecontrol system of claim 1, wherein said optical system includes aplurality of image array sensors.
 12. A method for automaticallycontrolling headlights of a vehicle, comprising: imaging externalsources of light within a predetermined field of view to obtain imagesof an environment outside of the vehicle; processing the obtained imagesfrom the optical system; and providing a control signal for controllingthe headlights as a function of the processed images.
 13. The method ofclaim 12, wherein the step of processing the obtained images comprisesidentifying a source of radiation in the images, and the step ofproviding the control signal comprises providing a control signal to dimthe headlights when a source of radiation in the images is identified asheadlight or taillight of another vehicle.
 14. The method of claim 12,wherein the step of processing the images comprises training a patternrecognition algorithm to identify the source of radiation in a trainingstage in which known sources of radiation are provided, images includingthese known sources are obtained and the pattern recognition is formedbased on the association of the known sources with the obtained images.15. The method of claim 14, wherein the pattern recognition algorithm isa neural network.
 16. The method of claim 12, further comprisingarranging an optical system on a rear view mirror in an interior of thevehicle to image the external sources of light.
 17. The method of claim12, further comprising arranging an optical system on a part of thevehicle that is not movable relative to a frame of the vehicle to imagethe external sources of light.
 18. The method of claim 12, wherein thestep of imaging the external sources of light comprises arranging animage array sensor containing a plurality of pixels on the vehicle. 19.The method of claim 12, wherein the step of imaging the external sourcesof light comprises arranging a plurality of image array sensors on thevehicle.
 20. In a vehicle including headlights and a rear view mirrorsituated inside a passenger compartment of the vehicle, a control systemfor automatically controlling the headlights of a vehicle, comprising:an optical system for imaging external sources of light, said opticalsystem being arranged on said rear view mirror; an image processingsystem for receiving images from said optical system and identifyingwhether sources of light in the images originate from headlights ortaillights of other vehicles, said image processing system beingarranged to dim the headlights when it identifies that a source of lightin an image originates from the headlights or taillights of anothervehicle.